09/01/2025 01:59:34.447 7f39255dd040 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:59:34.447 7f39255dd040 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:59:34.447 7f39255dd040 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:59:34.453 7f39255dd040 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/01/2025 01:59:34.453 7f39216cbd60 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/01/2025 01:59:34.453 7f39216cbd60 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/01/2025 01:59:34.454 7f39216efc20 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:59:34.454 7f39216efc20 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:59:34.454 7f39216efc20 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:59:34.454 7f39216efc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:59:34.454 7f39216efc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.455 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:34.455 7f39216efc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.455 7f39216efc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.455 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/01/2025 01:59:34.455 7f39216efc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.455 7f39216efc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.457 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.254:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:59:34.457 7f39216efc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.457 7f39216efc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.457 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.254:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:59:34.457 7f39216efc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.457 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:59:34.458 7f3921632540 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/01/2025 01:59:34.458 7f3921697d60 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/01/2025 01:59:34.459 7f39216440e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.459 7f39216440e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:59:34.459 7f39216440e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.459 7f39216440e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.459 7f39216440e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.254:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:59:34.459 7f39216440e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.510 7f39216efc20 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.254:5673//'] 09/01/2025 01:59:34.517 7f39216efc20 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:59:34.518 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.518 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.254:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:34.518 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.519 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.519 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.254:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:34.519 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.519 7f3921644ae0 [kazoo.client] [INFO]: Connecting to 10.0.0.254(10.0.0.254):2181, use_ssl: False 09/01/2025 01:59:34.520 7f3921644ae0 [kazoo.client] [DEBUG]: Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=60000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) 09/01/2025 01:59:34.520 7f3921644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.521 7f3921644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:34.521 7f3921644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.524 7f3921644ae0 [kazoo.client] [INFO]: Zookeeper connection established, state: CONNECTED 09/01/2025 01:59:34.525 7f3921644ae0 [contrail-alarm-gen] [INFO]: Analytics Discovery listen CONNECTED 09/01/2025 01:59:34.525 7f3921644ae0 [contrail-alarm-gen] [INFO]: Analytics Discovery to publish None 09/01/2025 01:59:34.525 7f3921644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.525 7f3921644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper re-established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:34.525 7f3921644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.525 7f3921644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.525 7f3921644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:34.525 7f3921644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.526 7f3921644ae0 [kazoo.client] [DEBUG]: Sending request(xid=1): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:34.527 7f3921644ae0 [kazoo.client] [DEBUG]: Received response(xid=1): ZnodeStat(czxid=229, mzxid=229, ctime=1756691951627, mtime=1756691951627, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=229) 09/01/2025 01:59:34.527 7f3921644ae0 [kazoo.client] [DEBUG]: Sending request(xid=2): GetChildren(path='/analytics-discovery-/AlarmGenerator', watcher=>) 09/01/2025 01:59:34.528 7f3921644ae0 [kazoo.client] [DEBUG]: Received response(xid=2): [] 09/01/2025 01:59:34.528 7f3921697d60 [contrail-alarm-gen] [INFO]: Analytics Discovery Watcher AlarmGenerator Children [] 09/01/2025 01:59:34.529 7f3921644ae0 [kazoo.client] [DEBUG]: Sending request(xid=3): Exists(path='/analytics-discovery-/Collector', watcher=None) 09/01/2025 01:59:34.529 7f3921644ae0 [kazoo.client] [DEBUG]: Received response(xid=3): ZnodeStat(czxid=230, mzxid=230, ctime=1756691951634, mtime=1756691951634, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=230) 09/01/2025 01:59:34.530 7f39216efc20 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:59:34.530 7f3921644ae0 [kazoo.client] [DEBUG]: Sending request(xid=4): GetChildren(path='/analytics-discovery-/Collector', watcher=>) 09/01/2025 01:59:34.530 7f3921644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.531 7f3921644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:34.531 7f3921644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.532 7f3921644ae0 [kazoo.client] [DEBUG]: Received response(xid=4): [] 09/01/2025 01:59:34.540 7f3921697d60 [contrail-alarm-gen] [INFO]: Analytics Discovery AlarmGenerator reconnect 09/01/2025 01:59:34.540 7f3921697d60 [contrail-alarm-gen] [INFO]: ensure /analytics-discovery-/AlarmGenerator 09/01/2025 01:59:34.540 7f3921697d60 [contrail-alarm-gen] [INFO]: zk state CONNECTED (CONNECTED) 09/01/2025 01:59:34.540 7f3921644ae0 [kazoo.client] [DEBUG]: Sending request(xid=5): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:34.541 7f3921644ae0 [kazoo.client] [DEBUG]: Received response(xid=5): ZnodeStat(czxid=229, mzxid=229, ctime=1756691951627, mtime=1756691951627, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=229) 09/01/2025 01:59:34.542 7f3921697d60 [contrail-alarm-gen] [INFO]: check for /analytics-discovery-/AlarmGenerator/cn-jenkins-deploy-platform-ansible-os-3992-1.-0 09/01/2025 01:59:34.542 7f3921644ae0 [kazoo.client] [DEBUG]: Sending request(xid=6): Exists(path='/analytics-discovery-/AlarmGenerator/cn-jenkins-deploy-platform-ansible-os-3992-1.-0', watcher=None) 09/01/2025 01:59:34.543 7f3921644ae0 [kazoo.client] [DEBUG]: Sending request(xid=7): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:34.543 7f3921644ae0 [kazoo.client] [DEBUG]: Received response(xid=7): ZnodeStat(czxid=229, mzxid=229, ctime=1756691951627, mtime=1756691951627, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=229) 09/01/2025 01:59:34.544 7f3921644ae0 [kazoo.client] [DEBUG]: Sending request(xid=8): GetChildren(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:34.545 7f3921644ae0 [kazoo.client] [DEBUG]: Received response(xid=8): [] 09/01/2025 01:59:34.545 7f3921697d60 [contrail-alarm-gen] [INFO]: Discovery AG callback : dict_values([]) 09/01/2025 01:59:34.551 7f39216efc20 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:59:34.551 7f3921598e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.551 7f3921598e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:34.551 7f3921598e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.554 7f39216efc20 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:59:34.554 7f3921598e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.555 7f3921598e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:34.555 7f3921598e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.555 7f39216efc20 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/01/2025 01:59:34.555 7f39216efc20 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/01/2025 01:59:34.555 7f39216efc20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:59:34.556 7f3921598e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:59:34.556 7f3921547ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.556 7f3921547ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:34.556 7f3921547ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.556 7f3921547ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.556 7f3921547ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:34.556 7f3921547ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.559 7f39216efc20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:59:34.567 7f3921598e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:59:34.568 7f392155b220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.568 7f392155b220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:34.569 7f392155b220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.596 7f39216efc20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:59:34.598 7f392156fb80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.598 7f392156fb80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:59:34.598 7f392156fb80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.603 7f39216efc20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:59:34.603 7f392155b220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.603 7f392155b220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:34.603 7f392155b220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.607 7f39216efc20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:59:34.625 7f39216efc20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:59:34.626 7f392152da40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.627 7f392152da40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:34.627 7f392152da40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.671 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config config_type = global_system_config operation = UPDATE config = { uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"fq_name : ["default-global-system-config"]autonomous_system : 64512enable_4byte_as : falseconfig_version : "1.0"ibgp_auto_mesh : truemac_aging_time : 300igmp_enable : falseenable_security_policy_draft : falseid_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4103235205771446636, "uuid_lslong": 10323580598023415145}, "enable": true, "created": "2025-09-01T01:56:13.973072", "last_modified": "2025-09-01T01:57:42.849425", "description": null, "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3992-1"], "attr": null, "uuid": "30040b85-8dc7-4ad8-8f92-322b051dfe3e"}] } 09/01/2025 01:59:34.672 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-address-mismatch-compute config_type = alarm operation = UPDATE config = { uuid : "4191d5c1-ed9e-499e-9315-361b8eee6a6f"fq_name : ["default-global-system-config", "system-defined-address-mismatch-compute"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_ip_address", "operand2": {"uve_attribute": "VrouterAgent.control_ip", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4724792512980339102, "uuid_lslong": 10598436790076336751}, "enable": true, "created": "2025-09-01T01:56:22.928755", "last_modified": "2025-09-01T01:56:22.928755", "description": "Compute Node IP Address mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-address-mismatch-compute" } 09/01/2025 01:59:34.672 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-address-mismatch-control config_type = alarm operation = UPDATE config = { uuid : "b7eb1fdc-6386-406b-9696-cb024616a97c"fq_name : ["default-global-system-config", "system-defined-address-mismatch-control"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "not in", "operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 13252721359890825323, "uuid_lslong": 10851083562822052220}, "enable": true, "created": "2025-09-01T01:56:23.004550", "last_modified": "2025-09-01T01:56:23.004550", "description": "Control Node IP Address mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-address-mismatch-control" } 09/01/2025 01:59:34.673 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-bgp-connectivity config_type = alarm operation = UPDATE config = { uuid : "6ffe2c9f-05ed-41e8-8720-2ad2630af7db"fq_name : ["default-global-system-config", "system-defined-bgp-connectivity"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "BgpRouterState.num_up_bgp_peer", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "BgpRouterState.num_up_bgp_peer", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 8069936643805364712, "uuid_lslong": 9736829477468174299}, "enable": true, "created": "2025-09-01T01:56:23.084176", "last_modified": "2025-09-01T01:56:23.084176", "description": "BGP peer mismatch. Not enough BGP peers are up.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-bgp-connectivity" } 09/01/2025 01:59:34.673 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-bottle-request-size-limit config_type = alarm operation = UPDATE config = { uuid : "e8724cf1-72aa-4754-a6be-e4111936f22a"fq_name : ["default-global-system-config", "system-defined-bottle-request-size-limit"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["config-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "UveConfigReq.err_info.*.", "operand2": {"uve_attribute": null, "json_value": "True"}, "variables": []}]}]}id_perms : {"permissions": null, "uuid": {"uuid_mslong": 16749534564038887252, "uuid_lslong": 12015291617959670314}, "enable": null, "created": "2025-09-01T01:56:14.564067", "last_modified": "2025-09-01T01:56:14.564067", "description": "Bottle request size limit exceeded.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []} } 09/01/2025 01:59:34.674 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-conf-elements-inconsistency config_type = alarm operation = UPDATE config = { uuid : "05929939-e5f0-42ff-be39-ce0d92f97535"fq_name : ["default-global-system-config", "system-defined-conf-elements-inconsistency"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "size==", "operand1": "ContrailConfig.elements", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 401551790739309311, "uuid_lslong": 13707213498574533941}, "enable": true, "created": "2025-09-01T01:56:22.874935", "last_modified": "2025-09-01T01:56:22.874935", "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-conf-elements-inconsistency" } 09/01/2025 01:59:34.674 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-conf-incorrect config_type = alarm operation = UPDATE config = { uuid : "52f85b25-45df-4b91-b134-f2d598522a93"fq_name : ["default-global-system-config", "system-defined-conf-incorrect"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "ContrailConfig", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 5978628720978512785, "uuid_lslong": 12769097842699676307}, "enable": true, "created": "2025-09-01T01:56:23.171905", "last_modified": "2025-09-01T01:56:23.171905", "description": "ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-conf-incorrect" } 09/01/2025 01:59:34.674 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-core-files config_type = alarm operation = UPDATE config = { uuid : "756bb3b1-dfde-4aa3-becd-ff8d27a14cde"fq_name : ["default-global-system-config", "system-defined-core-files"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "NodeStatus.all_core_file_list", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "NodeStatus.all_core_file_list", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 8461053901492079267, "uuid_lslong": 13748926219152346334}, "enable": true, "created": "2025-09-01T01:56:24.243849", "last_modified": "2025-09-01T01:56:24.243849", "description": "A core file has been generated on the node.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-core-files" } 09/01/2025 01:59:34.675 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-disk-usage-critical config_type = alarm operation = UPDATE config = { uuid : "50b6e636-cd3e-4fdd-b17c-0715caf49f33"fq_name : ["default-global-system-config", "system-defined-disk-usage-critical"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": ">", "operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operand2": {"uve_attribute": null, "json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 5816089091841609693, "uuid_lslong": 12789104832006102835}, "enable": true, "created": "2025-09-01T01:56:23.334551", "last_modified": "2025-09-01T01:56:23.334551", "description": "Disk usage crosses critical threshold limit.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-disk-usage-critical" } 09/01/2025 01:59:34.675 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-disk-usage-high config_type = alarm operation = UPDATE config = { uuid : "03be3fdc-3b00-4220-96b9-31204b7be7e1"fq_name : ["default-global-system-config", "system-defined-disk-usage-high"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "range", "operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operand2": {"uve_attribute": null, "json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 269723242804036128, "uuid_lslong": 10860765991155787745}, "enable": true, "created": "2025-09-01T01:56:23.259561", "last_modified": "2025-09-01T01:56:23.259561", "description": "Disk usage crosses high threshold limit.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-disk-usage-high" } 09/01/2025 01:59:34.676 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-node-status config_type = alarm operation = UPDATE config = { uuid : "a081e2e4-107e-4c4e-8162-1762dd41d6a7"fq_name : ["default-global-system-config", "system-defined-node-status"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11565774787221277774, "uuid_lslong": 9323039891996661415}, "enable": true, "created": "2025-09-01T01:56:23.404571", "last_modified": "2025-09-01T01:56:23.404571", "description": "Node Failure. NodeStatus UVE not present.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-node-status" } 09/01/2025 01:59:34.676 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-package-version-mismatch config_type = alarm operation = UPDATE config = { uuid : "87537fda-5839-4e19-9609-1c6b7db9e367"fq_name : ["default-global-system-config", "system-defined-package-version-mismatch"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "NodeStatus.running_package_version", "operand2": {"uve_attribute": "NodeStatus.installed_package_version", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9751278193947004441, "uuid_lslong": 10811203628475999079}, "enable": true, "created": "2025-09-01T01:56:24.417640", "last_modified": "2025-09-01T01:56:24.417640", "description": "There is a mismatch between installed and running package version.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-package-version-mismatch" } 09/01/2025 01:59:34.676 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-partial-sysinfo config_type = alarm operation = UPDATE config = { uuid : "2a931807-be30-4aa4-8121-5ee10b2947e1"fq_name : ["default-global-system-config", "system-defined-partial-sysinfo"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.build_info", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3067822192704113316, "uuid_lslong": 9304822625772062689}, "enable": true, "created": "2025-09-01T01:56:23.469931", "last_modified": "2025-09-01T01:56:23.469931", "description": "System Info Incomplete.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-partial-sysinfo" } 09/01/2025 01:59:34.676 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks config_type = alarm operation = UPDATE config = { uuid : "e4dacf4c-8f4a-4b54-b2e6-c53d60f74d17"fq_name : ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["database-nodeconfig-database-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": ">=", "operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"uve_attribute": null, "json_value": "300"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16490720913298967380, "uuid_lslong": 12891207850805120279}, "enable": true, "created": "2025-09-01T01:56:24.333079", "last_modified": "2025-09-01T01:56:24.333079", "description": "Pending compaction tasks in cassandra crossed the configured threshold.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-pending-cassandra-compaction-tasks" } 09/01/2025 01:59:34.677 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-process-connectivity config_type = alarm operation = UPDATE config = { uuid : "7d4895c9-0200-45db-a3d5-07e2c14a607c"fq_name : ["default-global-system-config", "system-defined-process-connectivity"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.process_status", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "NodeStatus.process_status.state", "operand2": {"uve_attribute": null, "json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9027630143618696667, "uuid_lslong": 11805350668708503676}, "enable": true, "created": "2025-09-01T01:56:23.535766", "last_modified": "2025-09-01T01:56:23.535766", "description": "Process(es) reporting as non-functional.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-process-connectivity" } 09/01/2025 01:59:34.677 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-process-status config_type = alarm operation = UPDATE config = { uuid : "4246429e-a45c-41a9-8983-f4dad1458bce"fq_name : ["default-global-system-config", "system-defined-process-status"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.process_info", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "NodeStatus.process_info.process_state", "operand2": {"uve_attribute": null, "json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4775577704002765225, "uuid_lslong": 9909032825796266958}, "enable": true, "created": "2025-09-01T01:56:23.606325", "last_modified": "2025-09-01T01:56:23.606325", "description": "Process Failure.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-process-status" } 09/01/2025 01:59:34.677 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-prouter-connectivity config_type = alarm operation = UPDATE config = { uuid : "c17e9d25-cc71-4e0d-a10e-8fe696180333"fq_name : ["default-global-system-config", "system-defined-prouter-connectivity"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["prouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.connected_agent_list", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 13942754282054962701, "uuid_lslong": 11605371510303753011}, "enable": true, "created": "2025-09-01T01:56:23.686927", "last_modified": "2025-09-01T01:56:23.686927", "description": "Prouter connectivity to controlling tor agent does not exist we look for non-empty value for connected_agent_list", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-prouter-connectivity" } 09/01/2025 01:59:34.678 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-prouter-tsn-connectivity config_type = alarm operation = UPDATE config = { uuid : "984f965e-cabb-460a-9ce5-0f74a813c3b8"fq_name : ["default-global-system-config", "system-defined-prouter-tsn-connectivity"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["prouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.tsn_agent_list", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}, {"operation": "!=", "operand1": "ProuterData.gateway_mode", "operand2": {"uve_attribute": null, "json_value": "\"SERVER\""}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.tsn_agent_list", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": []}, {"operation": "==", "operand1": "ProuterData.gateway_mode", "operand2": {"uve_attribute": null, "json_value": "\"SERVER\""}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10975156150797551114, "uuid_lslong": 11305459433293988792}, "enable": true, "created": "2025-09-01T01:56:23.775191", "last_modified": "2025-09-01T01:56:23.775191", "description": "Prouter connectivity to controlling tsn agent does not exist we look for non-empty value for tsn_agent_list", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-prouter-tsn-connectivity" } 09/01/2025 01:59:34.678 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-storage-cluster-state config_type = alarm operation = UPDATE config = { uuid : "b06a9834-465e-46e2-9564-0df2ef563f33"fq_name : ["default-global-system-config", "system-defined-storage-cluster-state"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["storage-cluster"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "StorageCluster.info_stats.status", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12712140248492951266, "uuid_lslong": 10764744346370981683}, "enable": true, "created": "2025-09-01T01:56:23.911037", "last_modified": "2025-09-01T01:56:23.911037", "description": "Storage Cluster warning/errors.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-storage-cluster-state" } 09/01/2025 01:59:34.678 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-interface config_type = alarm operation = UPDATE config = { uuid : "ad924d75-8d04-4de5-b81e-33a347f71132"fq_name : ["default-global-system-config", "system-defined-vrouter-interface"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": ">=", "operand1": "VrouterAgent.down_interface_count", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12507144282433670629, "uuid_lslong": 13267098328660119858}, "enable": true, "created": "2025-09-01T01:56:24.005826", "last_modified": "2025-09-01T01:56:24.005826", "description": "Vrouter interface(s) down.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-interface" } 09/01/2025 01:59:34.678 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down config_type = alarm operation = UPDATE config = { uuid : "b0338751-14ad-4c04-a95e-1d191983b364"fq_name : ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.l3mh_phy_interface_down", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12696640556796562436, "uuid_lslong": 12204224033860072292}, "enable": true, "created": "2025-09-01T01:56:24.695200", "last_modified": "2025-09-01T01:56:24.695200", "description": "On l3 multihomed compute one or more physical interfaces are Inactive.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-l3mh-physical-interface-link-down" } 09/01/2025 01:59:34.678 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-limit-exceeded config_type = alarm operation = UPDATE config = { uuid : "4a718103-ee3f-4a9b-905c-09cc7803c047"fq_name : ["default-global-system-config", "system-defined-vrouter-limit-exceeded"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.res_limit", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 5364210485056981659, "uuid_lslong": 10402200013110493255}, "enable": true, "created": "2025-09-01T01:56:24.510146", "last_modified": "2025-09-01T01:56:24.510146", "description": "Agent resource usage exceeded configured watermark for resource.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-limit-exceeded" } 09/01/2025 01:59:34.679 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-table-limit-exceeded config_type = alarm operation = UPDATE config = { uuid : "67e26b77-8150-4e68-b5c9-6db034bd9b9c"fq_name : ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.res_table_limit", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7485663691657989736, "uuid_lslong": 13099121594750311324}, "enable": true, "created": "2025-09-01T01:56:24.608648", "last_modified": "2025-09-01T01:56:24.608648", "description": "Agent resource usage exceeded table size for resource in vrouter.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-table-limit-exceeded" } 09/01/2025 01:59:34.679 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-xmpp-close-reason config_type = alarm operation = UPDATE config = { uuid : "45e9afa9-9efd-4204-9709-b90ae0c3a501"fq_name : ["default-global-system-config", "system-defined-xmpp-close-reason"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["xmpp-peer"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "XmppPeerInfoData.close_reason", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "!=", "operand1": "XmppPeerInfoData.state_info.state", "operand2": {"uve_attribute": null, "json_value": "\"Established\""}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 5037750801242341892, "uuid_lslong": 10883433430889243905}, "enable": true, "created": "2025-09-01T01:56:24.171114", "last_modified": "2025-09-01T01:56:24.171114", "description": "XMPP connection closed towards peer, alarm has close reason", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-xmpp-close-reason" } 09/01/2025 01:59:34.679 7f39216efc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-xmpp-connectivity config_type = alarm operation = UPDATE config = { uuid : "d0d6eecf-0cbd-46e2-934a-12f3224c8ccc"fq_name : ["default-global-system-config", "system-defined-xmpp-connectivity"]parent_type : "global-system-config"parent_uuid : "38f19eae-05e2-496c-8f44-b9db734ce569"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "BgpRouterState.num_up_xmpp_peer", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "BgpRouterState.num_up_xmpp_peer", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15048477777944463074, "uuid_lslong": 10613316307313790156}, "enable": true, "created": "2025-09-01T01:56:24.085409", "last_modified": "2025-09-01T01:56:24.085409", "description": "XMPP peer mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-xmpp-connectivity" } 09/01/2025 01:59:35.459 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:36.461 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:37.463 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:38.455 7f3920f23f40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:38.456 7f3920f23f40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:38.457 7f3920f23f40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:38.457 7f3920f23f40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:38.457 7f3920f23f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:38.457 7f3920f23f40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:38.457 7f3920f23f40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:38.457 7f3920f23f40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:38.458 7f3920f23f40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.458 7f3920f23f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.254:6379 server_addrs = [ 10.0.0.254:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:38.458 7f3920f23f40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.458 7f3920f23f40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.458 7f3920f23f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:38.458 7f3920f23f40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.464 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:39.465 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:39.521 7f3921632400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.522 7f3921632400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:39.522 7f3921632400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:40.467 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:41.469 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:42.457 7f3921632400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:42.459 7f3921632400 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:42.460 7f3921632400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:42.460 7f3921632400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:42.460 7f3921632400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:42.460 7f3921632400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:42.461 7f3921632400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:42.461 7f3921632400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:42.462 7f3921632400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.462 7f3921632400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:42.462 7f3921632400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.462 7f3921632400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.463 7f3921632400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:42.463 7f3921632400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.471 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:42.560 7f3921547a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3992-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1.'], 'id_perms': {'created': '2025-09-01T01:59:42.551873', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:42.551873', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12682759219656008150, 'uuid_mslong': 2370119825507566310}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '20e45b6c-827d-42e6-b002-36506b6821d6'}, 'oper': 'CREATE', 'request-id': 'req-cc1fd9bc-3d21-4140-9d67-d1085a4f2f87', 'type': 'analytics_snmp_node', 'uuid': '20e45b6c-827d-42e6-b002-36506b6821d6'} 09/01/2025 01:59:42.561 7f3921547a40 [contrail-alarm-gen] [ERROR]: Error in rabbitmq drainer greenlet: 'AnalyticsLogger' object has no attribute 'warn' 09/01/2025 01:59:42.561 7f3921547a40 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:59:42.562 7f3921632400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.562 7f3921632400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:42.562 7f3921632400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.569 7f3921547a40 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:59:42.569 7f3920f23a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.569 7f3920f23a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:42.569 7f3920f23a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.473 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:44.475 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:45.477 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:46.462 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:46.464 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:46.464 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:46.464 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:46.465 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:46.465 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:46.465 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:46.466 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:46.466 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.467 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:46.467 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.467 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.468 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:46.468 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.478 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:47.480 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:48.481 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:49.483 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:50.465 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:50.467 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:50.467 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:50.467 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:50.468 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:50.468 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:50.468 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:50.469 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:50.469 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.469 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:50.469 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.469 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.470 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:50.470 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.485 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:51.486 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:52.488 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:53.489 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:54.468 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:54.470 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:54.471 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:54.471 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:54.472 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:54.472 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:54.472 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:54.473 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:54.473 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.473 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:54.474 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.474 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.474 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:54.474 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.491 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:55.493 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:56.495 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:57.496 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:58.472 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:58.474 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:58.474 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:58.474 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:58.474 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:58.475 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:58.475 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:58.475 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:58.475 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.476 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:58.476 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.476 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.476 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:58.476 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.498 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:59.499 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:00.500 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:01.501 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:02.475 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:02.477 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:02.478 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:02.478 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:02.479 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:02.479 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:02.479 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:02.480 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:02.480 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.480 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:02.480 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.481 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.481 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:02.481 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.502 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:03.504 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:04.506 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:05.509 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:06.478 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:06.479 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:06.479 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:06.479 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:06.479 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:06.480 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:06.480 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:06.480 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:06.480 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.480 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:06.480 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.480 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.481 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:06.481 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.510 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:07.512 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:08.513 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:09.515 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:10.481 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:10.483 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:10.484 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:10.484 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:10.484 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:10.485 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:10.485 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:10.486 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:10.486 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.486 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:10.487 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.487 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.487 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:10.487 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.517 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:11.518 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:12.521 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:13.524 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:14.485 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:14.487 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:14.487 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:14.487 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:14.487 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:14.488 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:14.488 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:14.488 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:14.489 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.489 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:14.489 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.489 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.489 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:14.490 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.526 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:15.528 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:16.530 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:17.532 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:18.488 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:18.489 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:18.489 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:18.489 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:18.490 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:18.490 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:18.490 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:18.490 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:18.490 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.490 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:18.491 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.491 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.491 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:18.491 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.533 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:19.535 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:20.536 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:21.538 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:22.491 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:22.492 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:22.492 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:22.492 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:22.492 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:22.493 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:22.493 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:22.493 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:22.493 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.493 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:22.493 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.494 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.494 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:22.494 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.541 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:23.543 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:24.545 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:25.547 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:26.494 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:26.495 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:26.496 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:26.496 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:26.496 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:26.496 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:26.496 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:26.496 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:26.496 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.497 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:26.497 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.497 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.498 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:26.498 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.549 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:27.549 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:28.551 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:29.552 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:30.497 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:30.498 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:30.498 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:30.498 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:30.498 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:30.499 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:30.499 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:30.499 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:30.499 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.499 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:30.499 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.499 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.499 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:30.500 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.554 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:31.556 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:32.559 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:33.561 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:34.459 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:00:34.461 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.462 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:00:34.462 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.499 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:34.501 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:34.501 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:34.501 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:34.501 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:34.502 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:34.502 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:34.502 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:34.502 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.503 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:34.503 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.503 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.503 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:34.503 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.562 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:35.563 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:36.565 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:37.568 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:38.502 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:38.505 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:38.505 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:38.505 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:38.506 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:38.506 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:38.506 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:38.507 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:38.507 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.508 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:38.508 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.508 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.508 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:38.508 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.570 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:39.571 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:40.573 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:41.575 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:42.505 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:42.507 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:42.507 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:42.507 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:42.507 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:42.507 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:42.507 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:42.508 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:42.508 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.508 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:42.508 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.508 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.508 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:42.508 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.578 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:43.580 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:44.581 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:45.582 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:46.508 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:46.510 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:46.511 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:46.511 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:46.511 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:46.512 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:46.512 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:46.513 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:46.513 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.513 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:46.514 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.514 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.514 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:46.514 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.584 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:47.586 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:48.588 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:49.590 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:50.512 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:50.514 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:50.515 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:50.515 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:50.516 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:50.516 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:50.517 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:50.518 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:50.518 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.519 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:50.519 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.520 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.520 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:50.520 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.592 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:51.595 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:52.597 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:53.600 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:54.516 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:54.518 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:54.519 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:54.519 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:54.519 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:54.520 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:54.520 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:54.521 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:54.521 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.521 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:54.522 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.522 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.522 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:54.522 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.602 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:55.603 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:56.604 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:57.606 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:58.521 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:58.523 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:58.524 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:58.524 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:58.524 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:58.524 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:58.525 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:58.526 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:58.526 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.526 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:58.526 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.527 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.527 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:58.527 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.608 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:59.609 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:00.610 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:01.612 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:02.525 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:02.526 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:02.526 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:02.526 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:02.526 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:02.526 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:02.526 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:02.527 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:02.527 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.527 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:02.527 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.527 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.527 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:02.527 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.613 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:03.615 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:04.617 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:05.620 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:06.526 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:06.528 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:06.528 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:06.528 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:06.529 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:06.529 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:06.529 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:06.530 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:06.530 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.530 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:06.530 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.530 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.530 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:06.530 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.623 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:07.625 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:08.627 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:09.630 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:10.530 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:10.532 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:10.532 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:10.532 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:10.533 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:10.533 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:10.533 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:10.534 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:10.534 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.535 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:10.535 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.535 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.535 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:10.536 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.631 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:11.633 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:12.635 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:13.637 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:14.533 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:14.535 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:14.536 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:14.536 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:14.536 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:14.536 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:14.537 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:14.538 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:14.538 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.538 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:14.538 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.539 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.539 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:14.539 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.639 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:15.640 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:16.641 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:17.644 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:18.537 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:18.539 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:18.540 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:18.540 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:18.540 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:18.540 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:18.541 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:18.541 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:18.542 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.542 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:18.542 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.542 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.543 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:18.543 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.646 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:19.647 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:20.649 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:21.650 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:22.541 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:22.542 7f3920f23c20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:22.542 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:22.542 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:22.542 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:22.542 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:22.542 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:22.543 7f3920f23c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:22.543 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.543 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:22.543 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.543 7f3920f23c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.543 7f3920f23c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:22.543 7f3920f23c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.651 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:23.653 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:24.656 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:25.657 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:26.543 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:26.546 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:26.546 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:26.546 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:26.547 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:26.547 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:26.547 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:26.548 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:26.548 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.549 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:26.549 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.549 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.549 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:26.549 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.661 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:27.662 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:28.665 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:29.666 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:30.548 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:30.550 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:30.551 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:30.551 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:30.552 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:30.552 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:30.553 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:30.554 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:30.554 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.555 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:30.555 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.555 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.556 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:30.556 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.667 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:31.668 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:32.671 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:33.672 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:34.460 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:01:34.460 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.461 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:01:34.461 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.551 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:34.553 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:34.553 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:34.554 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:34.554 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:34.554 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:34.555 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:34.556 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:34.556 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.556 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:34.556 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.557 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.557 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:34.557 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.674 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:35.677 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:36.678 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:37.680 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:38.554 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:38.556 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:38.556 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:38.557 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:38.557 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:38.557 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:38.558 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:38.558 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:38.559 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.559 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:38.559 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.560 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.560 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:38.560 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.682 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:39.683 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:40.686 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:41.688 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:42.558 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:42.561 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:42.561 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:42.561 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:42.562 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:42.562 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:42.562 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:42.563 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:42.563 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.564 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:42.564 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.564 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.565 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:42.565 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.689 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:43.691 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:44.692 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:45.693 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:46.562 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:46.564 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:46.565 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:46.565 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:46.565 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:46.566 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:46.566 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:46.567 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:46.567 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.567 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:46.568 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.568 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.568 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:46.568 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.695 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:47.696 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:48.699 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:49.701 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:50.565 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:50.568 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:50.569 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:50.569 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:50.570 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:50.570 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:50.570 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:50.572 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:50.572 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.573 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:50.573 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.573 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.574 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:50.574 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.702 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:51.704 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:52.705 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:53.707 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:54.569 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:54.572 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:54.572 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:54.573 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:54.573 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:54.573 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:54.574 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:54.574 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:54.575 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.575 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:54.575 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.576 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.576 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:54.576 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.709 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:55.711 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:56.713 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:57.715 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:58.573 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:58.576 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:58.576 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:58.576 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:58.577 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:58.577 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:58.577 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:58.578 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:58.578 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.579 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:58.579 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.579 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.580 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:58.580 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.716 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:59.718 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:00.720 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:01.722 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:02.578 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:02.580 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:02.580 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:02.580 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:02.581 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:02.581 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:02.581 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:02.582 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:02.582 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.582 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:02.582 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.582 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.583 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:02.583 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.723 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:03.725 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:04.727 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:05.728 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:06.581 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:06.584 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:06.584 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:06.584 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:06.585 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:06.585 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:06.585 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:06.586 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:06.587 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.587 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:06.587 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.587 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.588 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:06.588 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.730 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:07.732 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:08.734 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:09.736 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:10.585 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:10.587 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:10.588 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:10.588 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:10.588 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:10.588 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:10.589 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:10.589 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:10.590 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.590 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:10.590 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.590 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.591 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:10.591 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.737 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:11.739 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:12.741 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:13.743 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:14.587 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:14.589 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:14.589 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:14.589 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:14.589 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:14.589 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:14.589 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:14.590 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:14.590 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.590 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:14.590 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.590 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.590 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:14.590 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.744 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:15.746 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:16.748 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:17.750 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:18.590 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:18.592 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:18.592 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:18.593 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:18.593 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:18.593 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:18.593 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:18.594 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:18.594 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.594 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:18.595 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.595 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.595 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:18.595 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.751 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:19.752 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:20.755 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:21.757 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:22.594 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:22.596 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:22.596 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:22.596 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:22.597 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:22.597 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:22.597 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:22.598 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:22.598 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.599 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:22.599 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.599 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.600 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:22.600 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.758 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:23.760 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:24.761 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:25.763 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:26.597 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:26.600 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:26.600 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:26.600 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:26.601 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:26.601 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:26.601 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:26.602 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:26.602 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.603 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:26.603 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.603 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.603 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:26.604 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.764 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:27.766 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:28.334 7f3921547a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-72faeda3-d2b0-4e32-99fc-270f9df96566', 'type': 'global_system_config', 'uuid': '38f19eae-05e2-496c-8f44-b9db734ce569'} 09/01/2025 02:02:28.487 7f3921547a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1'], 'obj_dict': {'display_name': 'cn-jenkins-deploy-platform-ansible-os-3992-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1'], 'id_perms': {'created': '2025-09-01T02:02:28.460717', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:28.460717', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11539493739750933229, 'uuid_mslong': 13298779682352221738}}, 'parent_type': 'global-system-config', 'parent_uuid': '38f19eae-05e2-496c-8f44-b9db734ce569', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b88ec1a9-40c9-4e2a-a024-846addb2c2ed', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.17'}, 'oper': 'CREATE', 'request-id': 'req-82b84dd8-7f4e-413d-b483-2116dba14f65', 'type': 'virtual_router', 'uuid': 'b88ec1a9-40c9-4e2a-a024-846addb2c2ed'} 09/01/2025 02:02:28.768 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:28.864 7f3921547a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:02:28.846248', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:28.846248', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9876995726292915272, 'uuid_mslong': 118707417928581946}}, 'parent_type': 'virtual-router', 'parent_uuid': 'b88ec1a9-40c9-4e2a-a024-846addb2c2ed', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '01a5bbc4-85d6-433a-8912-234851bd9448', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:01:a5:bb:c4:85']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '5e021764-73b8-4f96-ab85-adf191959170'}]}, 'oper': 'CREATE', 'request-id': 'req-4a51e9f4-714f-49fc-a54a-e7ab046530de', 'type': 'virtual_machine_interface', 'uuid': '01a5bbc4-85d6-433a-8912-234851bd9448'} 09/01/2025 02:02:28.890 7f3921547a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3992-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-4a51e9f4-714f-49fc-a54a-e7ab046530de', 'type': 'virtual_machine_interface', 'uuid': '01a5bbc4-85d6-433a-8912-234851bd9448'} 09/01/2025 02:02:29.476 7f3921547a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-258ee4b7-b364-49f8-ace1-a85ef0cecfe7', 'type': 'access_control_list', 'uuid': 'c20be28c-da85-4180-8860-c5fe339e3f00'} 09/01/2025 02:02:29.770 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:30.601 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:30.603 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:30.604 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:30.604 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:30.604 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:30.605 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:30.605 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:30.606 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:30.607 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.607 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:30.607 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.608 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.608 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:30.608 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.770 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:31.773 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:32.775 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:33.779 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:34.460 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:02:34.461 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.461 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:02:34.461 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.603 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:34.605 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:34.605 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:34.605 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:34.606 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:34.606 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:34.606 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:34.607 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:34.607 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.607 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:34.607 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.607 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.608 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:34.608 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.780 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:35.782 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:36.785 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:37.786 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:38.606 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:38.608 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:38.609 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:38.609 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:38.609 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:38.610 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:38.610 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:38.611 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:38.611 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.611 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:38.611 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.612 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.613 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:38.613 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.788 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:39.791 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:40.692 7f392155b4a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:40.792 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:41.794 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:42.609 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:42.612 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:42.612 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:42.613 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:42.613 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:42.613 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:42.613 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:42.614 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:42.615 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.616 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:42.616 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.616 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.616 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:42.617 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.795 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:43.797 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:44.799 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:45.800 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:46.614 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:46.616 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:46.616 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:46.617 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:46.617 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:46.618 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:46.618 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:46.619 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:46.619 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.619 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:46.619 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.620 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.620 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:46.620 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.803 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:47.805 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:48.807 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:49.809 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:50.617 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:50.618 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:50.618 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:50.619 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:50.619 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:50.619 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:50.619 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:50.620 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:50.620 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.620 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:50.620 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.620 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.621 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:50.621 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.810 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:51.811 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:52.813 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:53.571 7f392155b4a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:53.815 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:54.619 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:54.626 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:54.626 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:54.626 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:54.627 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:54.627 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:54.627 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:54.628 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:54.628 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.629 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:54.629 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.629 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.629 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:54.629 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.817 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:55.819 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:56.821 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:57.823 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:58.628 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:58.630 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:58.630 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:58.631 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:58.631 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:58.631 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:58.632 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:58.633 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:58.633 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.634 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:58.634 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.634 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.634 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:58.635 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.825 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:59.827 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:00.829 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:01.831 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:02.630 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:02.632 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:02.633 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:02.633 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:02.634 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:02.634 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:02.634 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:02.635 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:02.636 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.636 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:02.636 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.636 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.637 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:02.637 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.834 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:03.836 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:04.838 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:05.854 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:06.592 7f392155b4a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:06.633 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:06.635 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:06.636 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:06.636 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:06.636 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:06.636 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:06.637 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:06.638 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:06.638 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.638 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:06.638 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.639 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.639 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:06.639 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.856 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:07.858 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:08.860 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:09.862 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:10.636 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:10.638 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:10.639 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:10.639 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:10.639 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:10.639 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:10.640 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:10.640 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:10.641 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.641 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:10.641 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.641 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.642 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:10.642 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.863 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:11.865 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:12.868 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:13.869 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:14.639 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:14.641 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:14.642 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:14.642 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:14.642 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:14.643 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:14.643 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:14.644 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:14.644 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.645 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:14.645 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.645 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.646 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:14.646 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.870 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:15.873 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:16.874 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:17.877 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:18.642 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:18.645 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:18.646 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:18.646 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:18.647 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:18.647 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:18.647 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:18.649 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:18.649 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.650 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:18.650 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.650 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.651 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:18.651 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.880 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:19.586 7f392155b4a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:19.882 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:20.884 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:21.885 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:22.646 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:22.648 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:22.648 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:22.648 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:22.649 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:22.649 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:22.649 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:22.649 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:22.649 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.650 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:22.650 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.650 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.650 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:22.650 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.888 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:23.890 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:24.892 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:25.893 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:26.649 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:26.651 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:26.652 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:26.652 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:26.652 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:26.653 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:26.653 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:26.654 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:26.655 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.655 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:26.655 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.655 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.656 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:26.656 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.896 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:27.898 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:28.900 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:29.901 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:30.652 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:30.654 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:30.654 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:30.654 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:30.655 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:30.655 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:30.655 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:30.656 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:30.657 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.657 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:30.657 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.657 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.658 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:30.658 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.902 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:31.905 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:32.448 7f392155b4a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:32.907 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:33.909 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:34.461 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:03:34.462 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.462 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:03:34.462 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.654 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:34.656 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:34.656 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:34.656 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:34.656 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:34.657 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:34.657 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:34.657 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:34.657 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.657 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:34.657 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.658 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.658 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:34.658 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.911 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:35.913 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:36.915 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:37.916 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:38.657 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:38.658 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:38.658 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:38.659 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:38.659 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:38.659 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:38.659 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:38.659 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:38.659 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.659 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:38.659 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.659 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.659 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:38.660 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.918 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:39.920 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:40.922 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:41.924 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:42.660 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:42.662 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:42.663 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:42.663 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:42.663 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:42.664 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:42.664 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:42.664 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:42.665 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.665 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:42.665 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.666 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.666 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:42.666 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.926 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:43.928 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:44.930 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:45.784 7f392155b4a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:45.931 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:46.664 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:46.666 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:46.667 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:46.667 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:46.668 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:46.668 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:46.669 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:46.670 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:46.670 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.671 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:46.671 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.671 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.671 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:46.672 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.932 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:47.934 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:48.936 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:49.938 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:50.667 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:50.669 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:50.670 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:50.670 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:50.670 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:50.670 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:50.670 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:50.670 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:50.671 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.671 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:50.671 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.671 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.671 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:50.671 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.940 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:51.942 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:52.944 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:53.946 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:54.670 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:54.672 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:54.672 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:54.673 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:54.673 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:54.673 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:54.674 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:54.674 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:54.675 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.675 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:54.675 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.675 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.676 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:54.676 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.948 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:55.951 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:56.952 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:57.955 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:58.575 7f392155b4a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:58.673 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:58.674 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:58.674 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:58.674 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:58.674 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:58.674 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:58.674 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:58.675 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:58.675 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.675 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:58.675 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.675 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.676 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:58.676 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.956 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:59.959 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:00.961 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:01.963 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:02.674 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:02.676 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:02.676 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:02.676 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:02.676 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:02.677 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:02.677 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:02.677 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:02.678 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.678 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:02.678 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.678 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.678 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:02.678 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.965 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:03.968 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:04.969 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:05.971 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:06.677 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:06.679 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:06.680 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:06.680 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:06.681 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:06.681 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:06.681 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:06.682 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:06.682 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.683 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:06.683 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.683 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.683 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:06.684 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.973 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:07.975 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:08.977 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:09.978 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:10.680 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:10.681 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:10.682 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:10.682 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:10.682 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:10.682 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:10.682 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:10.682 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:10.682 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.682 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:10.682 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.683 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.683 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:10.683 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.980 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:11.564 7f392155b4a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:11.983 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:12.985 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:13.986 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:14.683 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:14.685 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:14.685 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:14.686 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:14.686 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:14.686 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:14.686 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:14.687 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:14.688 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.688 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:14.688 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.688 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.689 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:14.689 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.989 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:15.991 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:16.993 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:17.995 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:18.686 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:18.690 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:18.690 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:18.690 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:18.691 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:18.691 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:18.691 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:18.692 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:18.693 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.693 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:18.693 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.693 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.694 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:18.694 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.997 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:19.999 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:21.001 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:22.003 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:22.693 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:22.696 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:22.696 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:22.696 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:22.697 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:22.697 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:22.697 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:22.698 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:22.698 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.698 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:22.699 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.699 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.699 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:22.699 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:23.004 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:24.007 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:24.381 7f392155b4a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:25.009 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:26.012 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:26.696 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:26.699 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:26.699 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:26.699 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:26.700 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:26.700 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:26.700 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:26.701 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:26.701 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.701 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:26.702 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.702 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.702 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:26.702 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:27.014 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:28.016 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:29.018 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:30.020 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:30.700 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:30.702 7f392155b4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:30.703 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:30.703 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:30.703 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:30.704 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:30.704 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:30.705 7f392155b4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:30.705 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.706 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:30.706 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.706 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.706 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:30.707 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:31.022 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:32.025 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:33.027 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:34.028 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:34.459 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Starting the Kafka producer 09/01/2025 02:04:34.461 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name connections-closed 09/01/2025 02:04:34.461 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name connections-created 09/01/2025 02:04:34.462 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name select-time 09/01/2025 02:04:34.462 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name io-time 09/01/2025 02:04:34.463 7f3921632cc0 [kafka.client] [DEBUG]: Attempting to check version with node bootstrap-0 09/01/2025 02:04:34.463 7f3921632cc0 [kafka.client] [DEBUG]: Initiating connection to node bootstrap-0 at 10.0.0.254:9092 09/01/2025 02:04:34.463 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-sent-received 09/01/2025 02:04:34.464 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-sent 09/01/2025 02:04:34.464 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-received 09/01/2025 02:04:34.464 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name request-latency 09/01/2025 02:04:34.465 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name throttle-time 09/01/2025 02:04:34.465 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.bytes-sent 09/01/2025 02:04:34.466 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.bytes-received 09/01/2025 02:04:34.466 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.latency 09/01/2025 02:04:34.467 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.throttle 09/01/2025 02:04:34.467 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:04:34.468 7f392155b4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.469 7f392155b4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:04:34.469 7f392155b4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.469 7f3921632cc0 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:04:34.469 7f3921632cc0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:04:34.469 7f3921632cc0 [kafka.conn] [INFO]: [IPv4 ('10.0.0.254', 9092)]>: connecting to 10.0.0.254:9092 [('10.0.0.254', 9092) IPv4] 09/01/2025 02:04:34.470 7f3921632cc0 [kafka.client] [DEBUG]: Timeouts: user 199.962378, metadata inf, idle connection inf, request inf 09/01/2025 02:04:34.470 7f3921632cc0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: established TCP connection 09/01/2025 02:04:34.470 7f3921632cc0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: checking broker Api Versions 09/01/2025 02:04:34.470 7f3921632cc0 [kafka.protocol.parser] [DEBUG]: Sending request ApiVersionsRequest_v4(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:34.471 7f3921632cc0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 1 (timeout_ms 1600.0): ApiVersionsRequest_v4(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:34.471 7f3921632cc0 [kafka.client] [DEBUG]: Timeouts: user 199.978828, metadata inf, idle connection inf, request 1599.687338 09/01/2025 02:04:34.488 7f3921632cc0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:04:34.488 7f3921632cc0 [kafka.protocol.parser] [DEBUG]: Processing response ApiVersionsResponse_v4 09/01/2025 02:04:34.488 7f3921632cc0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 1 (17.62866973876953 ms): ApiVersionsResponse_v0(error_code=35, api_versions=[(api_key=18, min_version=0, max_version=3)]) 09/01/2025 02:04:34.489 7f3921632cc0 [kafka.client] [DEBUG]: Timeouts: user 199.964046, metadata inf, idle connection 539971.000000, request inf 09/01/2025 02:04:34.489 7f3921632cc0 [kafka.protocol.parser] [DEBUG]: Sending request ApiVersionsRequest_v3(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:34.489 7f3921632cc0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 2 (timeout_ms 1600.0): ApiVersionsRequest_v3(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:34.490 7f3921632cc0 [kafka.client] [DEBUG]: Timeouts: user 199.964046, metadata inf, idle connection 539970.000000, request 1599.644423 09/01/2025 02:04:34.500 7f3921632cc0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:04:34.500 7f3921632cc0 [kafka.protocol.parser] [DEBUG]: Processing response ApiVersionsResponse_v3 09/01/2025 02:04:34.500 7f3921632cc0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 2 (10.67662239074707 ms): ApiVersionsResponse_v3(error_code=0, api_versions=[(api_key=0, min_version=0, max_version=8, _tagged_fields={}), (api_key=1, min_version=0, max_version=11, _tagged_fields={}), (api_key=2, min_version=0, max_version=5, _tagged_fields={}), (api_key=3, min_version=0, max_version=9, _tagged_fields={}), (api_key=4, min_version=0, max_version=4, _tagged_fields={}), (api_key=5, min_version=0, max_version=3, _tagged_fields={}), (api_key=6, min_version=0, max_version=6, _tagged_fields={}), (api_key=7, min_version=0, max_version=3, _tagged_fields={}), (api_key=8, min_version=0, max_version=8, _tagged_fields={}), (api_key=9, min_version=0, max_version=7, _tagged_fields={}), (api_key=10, min_version=0, max_version=3, _tagged_fields={}), (api_key=11, min_version=0, max_version=7, _tagged_fields={}), (api_key=12, min_version=0, max_version=4, _tagged_fields={}), (api_key=13, min_version=0, max_version=4, _tagged_fields={}), (api_key=14, min_version=0, max_version=5, _tagged_fields={}), (api_key=15, min_version=0, max_version=5, _tagged_fields={}), (api_key=16, min_version=0, max_version=4, _tagged_fields={}), (api_key=17, min_version=0, max_version=1, _tagged_fields={}), (api_key=18, min_version=0, max_version=3, _tagged_fields={}), (api_key=19, min_version=0, max_version=5, _tagged_fields={}), (api_key=20, min_version=0, max_version=4, _tagged_fields={}), (api_key=21, min_version=0, max_version=2, _tagged_fields={}), (api_key=22, min_version=0, max_version=3, _tagged_fields={}), (api_key=23, min_version=0, max_version=3, _tagged_fields={}), (api_key=24, min_version=0, max_version=1, _tagged_fields={}), (api_key=25, min_version=0, max_version=1, _tagged_fields={}), (api_key=26, min_version=0, max_version=1, _tagged_fields={}), (api_key=27, min_version=0, max_version=0, _tagged_fields={}), (api_key=28, min_version=0, max_version=3, _tagged_fields={}), (api_key=29, min_version=0, max_version=2, _tagged_fields={}), (api_key=30, min_version=0, max_version=2, _tagged_fields={}), (api_key=31, min_version=0, max_version=2, _tagged_fields={}), (api_key=32, min_version=0, max_version=3, _tagged_fields={}), (api_key=33, min_version=0, max_version=1, _tagged_fields={}), (api_key=34, min_version=0, max_version=1, _tagged_fields={}), (api_key=35, min_version=0, max_version=2, _tagged_fields={}), (api_key=36, min_version=0, max_version=2, _tagged_fields={}), (api_key=37, min_version=0, max_version=2, _tagged_fields={}), (api_key=38, min_version=0, max_version=2, _tagged_fields={}), (api_key=39, min_version=0, max_version=2, _tagged_fields={}), (api_key=40, min_version=0, max_version=2, _tagged_fields={}), (api_key=41, min_version=0, max_version=2, _tagged_fields={}), (api_key=42, min_version=0, max_version=2, _tagged_fields={}), (api_key=43, min_version=0, max_version=2, _tagged_fields={}), (api_key=44, min_version=0, max_version=1, _tagged_fields={}), (api_key=45, min_version=0, max_version=0, _tagged_fields={}), (api_key=46, min_version=0, max_version=0, _tagged_fields={}), (api_key=47, min_version=0, max_version=0, _tagged_fields={}), (api_key=48, min_version=0, max_version=0, _tagged_fields={}), (api_key=49, min_version=0, max_version=0, _tagged_fields={})], throttle_time_ms=0, _tagged_fields={}) 09/01/2025 02:04:34.501 7f3921632cc0 [kafka.conn] [INFO]: [IPv4 ('10.0.0.254', 9092)]>: Broker version identified as 2.6 09/01/2025 02:04:34.501 7f3921632cc0 [kafka.conn] [INFO]: [IPv4 ('10.0.0.254', 9092)]>: Connection complete. 09/01/2025 02:04:34.501 7f3921632cc0 [kafka.client] [DEBUG]: Node bootstrap-0 connected 09/01/2025 02:04:34.501 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name batch-size 09/01/2025 02:04:34.501 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name compression-rate 09/01/2025 02:04:34.502 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name queue-time 09/01/2025 02:04:34.502 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name records-per-request 09/01/2025 02:04:34.502 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes 09/01/2025 02:04:34.502 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name record-retries 09/01/2025 02:04:34.502 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name errors 09/01/2025 02:04:34.502 7f3921632cc0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name record-size-max 09/01/2025 02:04:34.502 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Starting Kafka producer I/O thread. 09/01/2025 02:04:34.502 7f392155b4a0 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) to node bootstrap-0 09/01/2025 02:04:34.502 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) 09/01/2025 02:04:34.502 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 3 (timeout_ms 30000): MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) 09/01/2025 02:04:34.503 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.639044, metadata 30000.000000, idle connection 539958.000000, request 29999.979019 09/01/2025 02:04:34.503 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Kafka producer started 09/01/2025 02:04:34.503 7f3921632cc0 [contrail-alarm-gen] [INFO]: Initialized health-check KafkaProducer 09/01/2025 02:04:34.503 7f3921632cc0 [contrail-alarm-gen] [INFO]: health-check KafkaProducer call returned 09/01/2025 02:04:34.503 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994287, metadata 30000.000000, idle connection 539957.000000, request 29999.554157 09/01/2025 02:04:34.524 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 02:04:34.524 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:04:34.524 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 3 (21.76380157470703 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.254', port=9092, rack=None)], cluster_id='J9jkOynITIuGnNhnlMXSzw', controller_id=1, topics=[]) 09/01/2025 02:04:34.525 7f392155b4a0 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 0, groups: 0) 09/01/2025 02:04:34.525 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990711, metadata 299999.772949, idle connection 539935.000000, request inf 09/01/2025 02:04:34.703 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:34.704 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:34.704 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:34.704 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:34.704 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:34.704 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:34.704 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:34.705 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:34.705 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.705 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:34.705 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.705 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.705 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:34.705 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:35.030 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:36.032 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:37.033 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:37.220 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:38.035 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:38.705 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:38.708 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:38.708 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:38.708 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:38.709 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:38.709 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:38.710 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:38.711 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:38.711 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.712 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:38.712 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.712 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.713 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:38.713 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:39.038 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:40.040 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:41.041 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:42.045 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:42.709 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:42.711 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:42.712 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:42.712 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:42.712 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:42.712 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:42.713 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:42.714 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:42.714 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.715 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:42.715 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.715 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.716 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:42.716 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:43.047 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:44.049 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:45.052 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:46.054 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:46.712 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:46.713 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:46.713 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:46.713 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:46.714 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:46.714 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:46.714 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:46.714 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:46.714 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.714 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:46.714 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.714 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.714 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:46.714 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:47.056 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:48.058 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:49.060 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:50.062 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:50.448 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:50.714 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:50.717 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:50.717 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:50.717 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:50.717 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:50.717 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:50.717 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:50.718 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:50.718 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.718 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:50.718 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.718 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.718 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:50.718 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:51.064 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:52.067 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:53.069 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:54.070 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:54.717 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:54.719 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:54.719 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:54.719 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:54.719 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:54.719 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:54.720 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:54.720 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:54.720 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.721 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:54.721 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.721 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.721 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:54.721 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:55.071 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:56.073 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:57.074 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:58.077 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:58.720 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:58.722 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:58.722 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:58.723 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:58.723 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:58.723 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:58.724 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:58.724 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:58.725 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.725 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:58.725 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.725 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.726 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:58.726 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:59.079 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:00.081 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:01.083 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:02.085 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:02.722 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:02.723 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:02.724 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:02.724 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:02.724 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:02.724 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:02.724 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:02.725 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:02.725 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.725 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:02.725 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.725 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.725 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:02.725 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:03.087 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:03.285 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:04.089 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:05.090 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:06.093 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:06.724 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:06.727 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:06.727 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:06.728 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:06.728 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:06.728 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:06.729 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:06.730 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:06.730 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.730 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:06.730 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.731 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.731 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:06.731 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:07.095 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:08.097 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:09.098 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:10.100 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:10.728 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:10.730 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:10.730 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:10.731 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:10.731 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:10.731 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:10.731 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:10.732 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:10.732 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.733 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:10.733 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.733 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.733 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:10.733 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:11.102 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:12.104 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:13.106 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:14.109 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:14.730 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:14.732 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:14.732 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:14.733 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:14.733 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:14.733 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:14.733 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:14.734 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:14.734 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.734 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:14.734 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.734 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.735 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:14.735 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:15.111 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:16.113 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:16.125 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:17.116 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:18.117 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:18.733 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:18.736 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:18.736 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:18.736 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:18.737 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:18.737 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:18.737 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:18.738 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:18.738 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.739 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:18.739 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.739 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.739 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:18.740 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:19.119 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:20.121 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:21.123 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:22.125 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:22.737 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:22.739 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:22.739 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:22.740 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:22.740 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:22.740 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:22.741 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:22.741 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:22.742 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.742 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:22.742 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.742 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.743 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:22.743 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:23.127 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:24.129 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:25.130 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:26.132 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:26.740 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:26.742 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:26.743 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:26.743 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:26.743 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:26.744 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:26.744 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:26.745 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:26.745 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.745 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:26.745 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.746 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.746 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:26.746 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:27.134 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:28.136 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:29.073 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:29.139 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:30.141 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:30.744 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:30.746 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:30.747 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:30.747 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:30.748 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:30.748 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:30.748 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:30.749 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:30.749 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.750 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:30.750 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.750 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.751 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:30.751 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:31.143 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:32.144 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:33.147 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:34.149 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:34.469 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:05:34.470 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.470 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:05:34.470 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.747 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:34.749 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:34.749 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:34.750 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:34.750 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:34.750 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:34.750 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:34.751 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:34.751 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.751 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:34.752 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.752 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.752 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:34.752 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:35.151 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:36.153 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:37.156 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:38.158 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:38.750 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:38.752 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:38.753 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:38.753 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:38.754 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:38.754 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:38.754 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:38.755 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:38.755 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.756 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:38.756 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.756 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.757 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:38.757 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:39.161 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:40.163 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:41.164 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:41.979 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:42.166 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:42.754 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:42.756 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:42.757 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:42.757 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:42.758 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:42.758 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:42.758 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:42.760 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:42.760 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.760 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:42.761 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.761 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.761 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:42.761 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:43.168 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:44.171 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:45.172 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:46.175 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:46.757 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:46.758 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:46.758 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:46.758 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:46.759 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:46.759 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:46.759 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:46.759 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:46.759 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.759 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:46.759 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.759 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.760 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:46.760 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:47.176 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:48.178 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:49.179 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:50.181 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:50.759 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:50.761 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:50.762 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:50.762 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:50.763 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:50.763 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:50.763 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:50.764 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:50.764 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.765 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:50.765 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.766 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.766 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:50.766 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:51.184 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:52.186 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:53.188 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:54.191 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:54.762 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:54.764 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:54.764 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:54.764 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:54.765 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:54.765 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:54.765 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:54.766 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:54.766 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.767 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:54.767 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.767 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.767 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:54.767 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:55.182 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:55.191 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:56.192 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:57.194 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:58.197 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:58.765 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:58.767 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:58.768 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:58.768 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:58.768 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:58.769 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:58.769 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:58.770 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:58.770 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.770 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:58.771 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.771 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.771 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:58.771 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:59.199 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:00.201 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:01.203 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:02.205 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:02.768 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:02.771 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:02.771 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:02.771 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:02.772 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:02.772 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:02.772 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:02.773 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:02.773 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.774 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:02.774 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.774 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.774 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:02.775 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:03.207 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:04.209 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:05.212 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:06.214 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:06.773 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:06.775 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:06.777 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:06.777 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:06.778 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:06.778 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:06.779 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:06.780 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:06.780 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.781 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:06.781 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.781 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.782 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:06.782 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:07.216 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:08.146 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:08.218 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:09.220 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:10.223 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:10.777 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:10.779 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:10.780 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:10.780 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:10.780 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:10.780 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:10.780 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:10.781 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:10.782 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.782 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:10.782 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.782 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.783 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:10.783 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:11.225 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:12.227 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:13.229 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:14.231 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:14.780 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:14.783 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:14.783 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:14.783 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:14.784 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:14.784 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:14.784 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:14.785 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:14.785 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.785 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:14.786 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.786 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.786 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:14.786 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:15.234 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:16.237 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:17.239 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:18.241 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:18.785 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:18.787 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:18.787 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:18.788 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:18.788 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:18.788 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:18.789 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:18.790 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:18.790 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.791 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:18.791 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.791 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.792 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:18.792 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:19.243 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:20.246 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:21.158 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:21.248 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:22.249 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:22.788 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:22.788 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:22.789 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:22.789 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:22.789 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:22.789 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:22.789 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:22.789 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:22.789 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.789 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:22.790 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.790 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.790 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:22.790 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:23.250 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:24.252 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:25.255 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:26.257 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:26.790 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:26.792 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:26.792 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:26.793 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:26.793 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:26.793 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:26.793 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:26.794 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:26.795 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.795 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:26.795 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.795 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.796 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:26.796 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:27.259 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:28.261 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:29.263 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:30.265 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:30.793 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:30.794 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:30.794 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:30.794 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:30.794 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:30.794 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:30.794 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:30.795 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:30.795 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.795 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:30.795 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.795 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.795 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:30.795 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:31.268 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:32.269 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:33.272 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:33.922 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:34.274 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:34.469 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:06:34.470 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.470 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:06:34.470 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.504 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:34.504 7f392155b4a0 [kafka.client] [DEBUG]: Initializing connection to node 1 for metadata request 09/01/2025 02:06:34.505 7f392155b4a0 [kafka.client] [DEBUG]: Initiating connection to node 1 at 10.0.0.254:9092 09/01/2025 02:06:34.505 7f392155b4a0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.bytes-sent 09/01/2025 02:06:34.505 7f392155b4a0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.bytes-received 09/01/2025 02:06:34.505 7f392155b4a0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.latency 09/01/2025 02:06:34.505 7f392155b4a0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.throttle 09/01/2025 02:06:34.506 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:06:34.506 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:06:34.506 7f392155b4a0 [kafka.conn] [INFO]: [IPv4 ('10.0.0.254', 9092)]>: connecting to 10.0.0.254:9092 [('10.0.0.254', 9092) IPv4] 09/01/2025 02:06:34.506 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999987.862825, metadata inf, idle connection 419954.000000, request inf 09/01/2025 02:06:34.507 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: established TCP connection 09/01/2025 02:06:34.507 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: checking broker Api Versions 09/01/2025 02:06:34.507 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Using pre-configured api_version (2, 6) for ApiVersions 09/01/2025 02:06:34.507 7f392155b4a0 [kafka.conn] [INFO]: [IPv4 ('10.0.0.254', 9092)]>: Connection complete. 09/01/2025 02:06:34.507 7f392155b4a0 [kafka.client] [DEBUG]: Node 1 connected 09/01/2025 02:06:34.507 7f392155b4a0 [kafka.conn] [INFO]: [IPv4 ('10.0.0.254', 9092)]>: Closing connection. 09/01/2025 02:06:34.507 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: reconnect backoff 0.051747972137624665 after 1 failures 09/01/2025 02:06:34.508 7f392155b4a0 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:06:34.508 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:34.508 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 1 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:34.508 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.447832, metadata 30000.000000, idle connection 419952.000000, request 29999.967098 09/01/2025 02:06:34.509 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990711, metadata 30000.000000, idle connection 419952.000000, request 29999.552727 09/01/2025 02:06:34.552 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:06:34.552 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:34.553 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 1 (44.284820556640625 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.254', port=9092, rack=None)], cluster_id='J9jkOynITIuGnNhnlMXSzw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:34.553 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991188, metadata 99.966553, idle connection 419907.000000, request inf 09/01/2025 02:06:34.553 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.04936528205871582 secs. 09/01/2025 02:06:34.553 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:34.553 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994049, metadata 99.445312, idle connection 419907.000000, request inf 09/01/2025 02:06:34.654 7f392155b4a0 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:06:34.654 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:34.654 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 2 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:34.654 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.363909, metadata 30000.000000, idle connection 419806.000000, request 29999.968529 09/01/2025 02:06:34.655 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991426, metadata 30000.000000, idle connection 419806.000000, request 29999.540091 09/01/2025 02:06:34.661 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:06:34.661 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:34.661 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 2 (6.782770156860352 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.254', port=9092, rack=None)], cluster_id='J9jkOynITIuGnNhnlMXSzw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:34.661 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991426, metadata 99.964600, idle connection 419799.000000, request inf 09/01/2025 02:06:34.661 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.15787768363952637 secs. 09/01/2025 02:06:34.662 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:34.662 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993334, metadata 99.500488, idle connection 419798.000000, request inf 09/01/2025 02:06:34.763 7f392155b4a0 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:06:34.763 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:34.764 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 3 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:34.764 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999988.484144, metadata 30000.000000, idle connection 419696.000000, request 29999.918222 09/01/2025 02:06:34.765 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.977360, metadata 30000.000000, idle connection 419695.000000, request 29998.837471 09/01/2025 02:06:34.775 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 02:06:34.775 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:34.776 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 3 (11.390924453735352 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.254', port=9092, rack=None)], cluster_id='J9jkOynITIuGnNhnlMXSzw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:34.776 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.977121, metadata 99.915039, idle connection 419684.000000, request inf 09/01/2025 02:06:34.777 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.27311134338378906 secs. 09/01/2025 02:06:34.777 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:34.778 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.979267, metadata 98.546875, idle connection 419683.000000, request inf 09/01/2025 02:06:34.794 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:34.797 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:34.798 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:34.798 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:34.799 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:34.799 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:34.799 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:34.801 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:34.801 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.802 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:34.802 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.802 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.803 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:34.803 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.878 7f392155b4a0 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:06:34.878 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:34.878 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 4 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:34.878 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.317179, metadata 30000.000000, idle connection 419582.000000, request 29999.966383 09/01/2025 02:06:34.879 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991665, metadata 30000.000000, idle connection 419581.000000, request 29999.508619 09/01/2025 02:06:34.886 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 4 09/01/2025 02:06:34.886 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:34.886 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 4 (7.910013198852539 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.254', port=9092, rack=None)], cluster_id='J9jkOynITIuGnNhnlMXSzw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:34.887 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990950, metadata 99.963623, idle connection 419574.000000, request inf 09/01/2025 02:06:34.887 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.3832583427429199 secs. 09/01/2025 02:06:34.887 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:34.887 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992142, metadata 99.362793, idle connection 419573.000000, request inf 09/01/2025 02:06:34.988 7f392155b4a0 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:06:34.988 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:34.988 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 5 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:34.988 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.434004, metadata 30000.000000, idle connection 419472.000000, request 29999.973536 09/01/2025 02:06:34.989 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993334, metadata 30000.000000, idle connection 419472.000000, request 29999.618292 09/01/2025 02:06:34.996 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 5 09/01/2025 02:06:34.996 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:34.996 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 5 (7.8411102294921875 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.254', port=9092, rack=None)], cluster_id='J9jkOynITIuGnNhnlMXSzw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:34.996 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992857, metadata 99.968018, idle connection 419464.000000, request inf 09/01/2025 02:06:34.997 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.49295806884765625 secs. 09/01/2025 02:06:34.997 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:34.997 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994049, metadata 99.595215, idle connection 419463.000000, request inf 09/01/2025 02:06:35.098 7f392155b4a0 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:06:35.098 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:35.099 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 6 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:35.099 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999988.554716, metadata 30000.000000, idle connection 419361.000000, request 29999.901056 09/01/2025 02:06:35.100 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.981651, metadata 30000.000000, idle connection 419360.000000, request 29998.934031 09/01/2025 02:06:35.115 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 6 09/01/2025 02:06:35.115 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:35.116 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 6 (16.633033752441406 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.254', port=9092, rack=None)], cluster_id='J9jkOynITIuGnNhnlMXSzw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:35.116 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.985228, metadata 99.916992, idle connection 419344.000000, request inf 09/01/2025 02:06:35.116 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.6128230094909668 secs. 09/01/2025 02:06:35.117 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:35.117 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.988327, metadata 99.128662, idle connection 419343.000000, request inf 09/01/2025 02:06:35.218 7f392155b4a0 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:06:35.218 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:35.219 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 7 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:35.219 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999988.643885, metadata 30000.000000, idle connection 419241.000000, request 29999.927521 09/01/2025 02:06:35.220 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978552, metadata 30000.000000, idle connection 419240.000000, request 29998.965740 09/01/2025 02:06:35.228 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 7 09/01/2025 02:06:35.228 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:35.228 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 7 (9.148359298706055 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.254', port=9092, rack=None)], cluster_id='J9jkOynITIuGnNhnlMXSzw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:35.230 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.984989, metadata 99.925049, idle connection 419230.000000, request inf 09/01/2025 02:06:35.230 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.7267887592315674 secs. 09/01/2025 02:06:35.231 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:35.231 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.987612, metadata 98.981934, idle connection 419229.000000, request inf 09/01/2025 02:06:35.275 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:35.332 7f392155b4a0 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:06:35.332 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:35.332 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 8 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:35.332 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.080429, metadata 30000.000000, idle connection 419128.000000, request 29999.953508 09/01/2025 02:06:35.333 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.987850, metadata 30000.000000, idle connection 419127.000000, request 29999.289989 09/01/2025 02:06:35.338 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 8 09/01/2025 02:06:35.338 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:35.339 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 8 (6.943941116333008 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.254', port=9092, rack=None)], cluster_id='J9jkOynITIuGnNhnlMXSzw', controller_id=1, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=5, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=10, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=20, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=25, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=26, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=11, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=4, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=16, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=17, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=23, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=28, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=13, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=22, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=8, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=2, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=29, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=19, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=14, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=1, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=7, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[])])]) 09/01/2025 02:06:35.341 7f392155b4a0 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 1, groups: 0) 09/01/2025 02:06:35.341 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.984751, metadata 299999.739746, idle connection 419119.000000, request inf 09/01/2025 02:06:35.342 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.8381013870239258 secs. 09/01/2025 02:06:35.342 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=7) 09/01/2025 02:06:35.342 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=7) is either full or getting a new batch 09/01/2025 02:06:35.343 7f3921632cc0 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:06:35.343 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:06:35.343 7f3921632cc0 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=7) 09/01/2025 02:06:35.344 7f392155b4a0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.records-per-batch 09/01/2025 02:06:35.344 7f392155b4a0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.bytes 09/01/2025 02:06:35.344 7f392155b4a0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.compression-rate 09/01/2025 02:06:35.344 7f392155b4a0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.record-retries 09/01/2025 02:06:35.344 7f392155b4a0 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.record-errors 09/01/2025 02:06:35.345 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:06:35.345 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x025\xe3A\xaf\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06\xb5N\x00\x00\x01\x99\x03\x06\xb5N\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:06:35.345 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x025\xe3A\xaf\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06\xb5N\x00\x00\x01\x99\x03\x06\xb5N\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:06:35.346 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x025\xe3A\xaf\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06\xb5N\x00\x00\x01\x99\x03\x06\xb5N\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:06:35.346 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 9 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x025\xe3A\xaf\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06\xb5N\x00\x00\x01\x99\x03\x06\xb5N\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:06:35.347 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 299994.699219, idle connection 419114.000000, request 29999.940157 09/01/2025 02:06:35.347 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.987612, metadata 299994.080811, idle connection 419113.000000, request 29999.327898 09/01/2025 02:06:35.440 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 9 09/01/2025 02:06:35.440 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:06:35.441 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 9 (94.35677528381348 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:06:35.442 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:06:35.442 7f392155b4a0 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=7) with base offset 0 log start offset 0. 09/01/2025 02:06:35.443 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.974737, metadata 299898.628174, idle connection 419018.000000, request inf 09/01/2025 02:06:35.443 7f3921632cc0 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:06:36.277 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:37.279 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:38.281 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:38.798 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:38.800 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:38.800 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:38.800 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:38.801 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:38.801 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:38.801 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:38.802 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:38.802 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.803 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:38.803 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.803 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.803 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:38.804 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:39.284 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:40.286 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:41.288 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:42.291 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:42.801 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:42.802 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:42.803 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:42.803 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.803 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:42.803 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.803 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:42.804 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:42.804 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.804 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:42.804 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.804 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.804 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:42.804 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:43.292 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:44.294 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:45.296 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:46.297 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:46.803 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:46.804 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:46.804 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:46.804 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.805 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:46.805 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.805 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:46.805 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:46.805 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.805 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:46.805 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.806 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.806 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:46.806 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.828 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:47.299 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:48.301 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:49.303 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:50.305 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:50.805 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:50.806 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:50.806 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:50.806 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.806 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:50.806 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.807 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:50.807 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:50.807 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.807 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:50.807 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.807 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.807 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:50.807 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:51.307 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:52.309 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:53.311 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:54.314 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:54.807 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:54.809 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:54.809 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:54.809 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.809 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:54.810 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.810 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:54.810 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:54.810 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.811 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:54.811 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.811 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.811 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:54.811 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:55.316 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:56.318 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:57.320 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:58.322 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:58.810 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:58.811 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:58.812 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:58.812 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.812 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:58.813 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.813 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:58.814 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:58.814 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.814 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:58.814 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.815 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.815 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:58.815 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:59.324 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:00.014 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:00.326 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:01.327 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:02.329 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:02.813 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:02.816 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:02.817 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:02.817 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.817 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:02.818 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.818 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:02.819 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:02.819 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.819 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:02.820 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.820 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.820 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:02.820 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:03.330 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:04.332 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:05.334 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:06.336 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:06.817 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:06.818 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:06.819 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:06.819 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.819 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:06.819 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.819 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:06.820 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:06.820 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.820 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:06.821 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.821 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.821 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:06.821 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:07.338 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:08.340 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:09.341 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:10.344 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:10.820 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:10.822 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:10.822 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:10.823 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.823 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:10.823 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.824 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:10.824 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:10.825 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.825 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:10.825 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.826 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.826 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:10.826 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:11.347 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:12.349 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:13.008 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:13.352 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:14.370 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:14.823 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:14.826 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:14.826 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:14.826 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.827 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:14.827 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.828 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:14.829 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:14.829 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.830 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:14.830 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.831 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.831 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:14.831 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:15.372 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:16.375 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:17.377 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:18.379 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:18.827 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:18.830 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:18.830 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:18.830 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.831 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:18.831 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.831 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:18.832 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:18.832 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.833 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:18.833 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.833 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.833 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:18.834 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:19.381 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:20.384 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:21.386 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:22.388 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:22.831 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:22.832 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:22.832 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:22.832 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.832 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:22.832 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.832 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:22.832 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:22.833 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.833 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:22.833 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.833 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.833 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:22.833 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:23.389 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:24.392 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:25.394 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:26.120 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:26.396 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:26.833 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:26.835 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:26.835 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:26.835 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:26.836 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:26.836 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:26.836 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:26.837 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:26.837 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.838 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:26.838 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.838 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.838 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:26.839 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:27.398 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:28.401 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:29.403 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:30.404 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:30.836 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:30.839 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:30.839 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:30.839 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:30.840 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:30.840 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:30.841 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:30.842 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:30.842 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.843 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:30.843 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.843 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.844 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:30.844 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.405 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:32.408 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:33.409 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:34.411 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:34.470 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:07:34.471 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.472 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:07:34.472 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.840 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:34.841 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:34.841 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:34.841 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:34.841 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:34.842 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:34.842 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:34.842 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:34.842 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.842 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:34.842 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.842 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.842 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:34.842 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.413 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:36.415 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:37.416 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:38.418 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:38.842 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:38.844 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:38.844 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:38.844 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:38.844 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:38.844 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:38.845 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:38.845 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:38.845 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.846 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:38.846 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.846 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.846 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:38.846 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.886 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:39.420 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:40.422 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:41.424 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:42.427 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:42.844 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:42.846 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:42.847 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:42.847 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:42.847 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:42.847 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:42.847 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:42.848 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:42.848 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.848 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:42.849 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.849 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.849 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:42.849 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.430 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:44.433 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:45.437 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:46.440 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:46.848 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:46.850 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:46.850 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:46.850 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:46.851 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:46.851 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:46.851 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:46.852 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:46.852 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.853 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:46.853 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.853 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.853 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:46.853 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.442 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:48.445 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:49.447 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:50.450 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:50.851 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:50.854 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:50.855 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:50.855 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:50.856 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:50.856 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:50.856 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:50.857 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:50.858 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.858 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:50.859 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.859 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.859 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:50.860 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.452 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:51.624 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:52.453 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:53.456 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:54.457 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:54.856 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:54.858 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:54.859 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:54.859 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:54.859 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:54.859 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:54.860 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:54.860 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:54.861 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.861 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:54.862 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.862 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.862 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:54.862 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.459 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:56.461 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:57.463 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:58.465 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:58.858 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:58.860 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:58.861 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:58.861 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:58.861 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:58.861 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:58.861 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:58.862 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:58.862 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.862 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:58.863 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.863 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.863 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:58.863 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.467 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:00.469 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:01.471 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:02.473 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:02.862 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:02.864 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:02.864 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:02.864 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:02.865 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:02.865 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:02.865 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:02.866 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:02.866 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.866 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:02.867 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.867 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.867 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:02.867 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.474 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:04.476 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:04.650 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:05.477 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:06.479 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:06.864 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:06.866 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:06.866 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:06.866 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:06.866 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:06.866 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:06.866 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:06.867 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:06.867 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.867 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:06.867 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.867 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.867 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:06.867 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.481 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:08.483 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:09.485 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:10.486 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:10.866 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:10.867 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.868 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:10.869 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.488 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:12.490 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:13.493 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:14.495 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:14.869 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:14.872 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:14.872 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:14.873 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:14.874 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:14.874 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:14.874 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:14.876 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:14.876 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.877 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:14.878 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.879 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.880 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:14.880 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.497 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:16.500 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:17.502 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:17.748 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:18.504 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:18.872 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:18.874 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:18.875 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:18.875 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:18.876 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:18.876 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:18.876 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:18.877 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:18.877 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.877 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:18.878 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.878 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.878 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:18.878 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.505 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:20.507 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:21.510 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:22.513 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:22.875 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:22.878 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:22.878 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:22.879 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:22.879 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:22.879 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:22.879 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:22.880 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:22.881 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.881 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:22.881 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.881 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.882 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:22.882 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.515 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:24.517 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:25.519 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:26.521 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:26.879 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:26.881 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:26.881 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:26.881 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:26.882 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:26.882 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:26.882 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:26.883 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:26.884 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.884 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:26.884 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.885 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.885 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:26.885 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.522 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:28.525 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:29.527 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:30.517 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:30.528 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:30.882 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:30.884 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:30.885 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:30.885 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:30.886 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:30.886 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:30.886 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:30.888 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:30.888 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.888 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:30.889 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.889 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.889 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:30.890 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.529 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:32.531 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:33.533 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:34.472 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:08:34.473 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.473 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:08:34.474 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.534 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:34.885 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:34.888 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:34.888 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:34.888 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:34.889 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:34.889 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:34.889 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:34.890 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:34.890 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.891 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:34.891 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.891 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.891 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:34.892 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.445 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=18) 09/01/2025 02:08:35.445 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=18) is either full or getting a new batch 09/01/2025 02:08:35.445 7f3921632cc0 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:08:35.445 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:08:35.445 7f3921632cc0 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=18) 09/01/2025 02:08:35.446 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:08:35.446 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xbe\x08t0\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08\x8au\x00\x00\x01\x99\x03\x08\x8au\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:08:35.446 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xbe\x08t0\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08\x8au\x00\x00\x01\x99\x03\x08\x8au\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:08:35.446 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xbe\x08t0\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08\x8au\x00\x00\x01\x99\x03\x08\x8au\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:08:35.446 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 10 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xbe\x08t0\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08\x8au\x00\x00\x01\x99\x03\x08\x8au\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:08:35.447 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 179894.590820, idle connection 299014.000000, request 29999.960184 09/01/2025 02:08:35.447 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992857, metadata 179894.123779, idle connection 299013.000000, request 29999.500513 09/01/2025 02:08:35.451 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 10 09/01/2025 02:08:35.451 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:08:35.451 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 10 (4.201650619506836 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:08:35.451 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:08:35.451 7f392155b4a0 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=18) with base offset 0 log start offset 0. 09/01/2025 02:08:35.451 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992380, metadata 179889.942383, idle connection 299009.000000, request inf 09/01/2025 02:08:35.451 7f3921632cc0 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:08:35.536 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:36.537 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:37.539 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:38.541 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:38.888 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:38.891 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:38.892 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:38.892 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:38.892 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:38.893 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:38.893 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:38.894 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:38.894 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.894 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:38.894 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.895 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.895 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:38.895 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.543 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:40.545 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:41.547 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:42.549 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:42.892 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:42.894 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:42.895 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:42.895 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:42.895 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:42.895 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:42.895 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:42.896 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:42.896 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.896 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:42.896 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.897 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.897 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:42.897 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.195 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:43.551 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:44.553 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:45.555 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:46.556 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:46.896 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:46.898 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:46.899 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:46.899 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:46.899 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:46.900 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:46.900 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:46.901 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:46.901 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.901 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:46.902 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.902 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.902 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:46.902 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.558 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:48.560 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:49.563 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:50.564 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:50.899 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:50.900 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:50.900 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:50.900 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:50.900 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:50.900 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:50.900 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:50.901 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:50.901 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.901 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:50.901 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.901 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.901 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:50.901 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.566 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:52.569 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:53.571 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:54.573 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:54.901 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:54.903 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:54.903 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:54.903 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:54.904 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:54.904 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:54.904 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:54.904 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:54.904 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.905 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:54.905 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.905 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.905 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:54.905 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.575 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:55.845 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:56.578 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:57.580 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:58.582 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:58.904 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:58.907 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:58.907 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:58.907 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:58.908 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:58.908 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:58.908 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:58.909 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:58.910 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.911 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:58.911 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.911 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.911 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:58.912 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.585 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:00.587 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:01.589 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:02.591 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:02.908 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:02.909 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:02.910 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:02.910 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:02.910 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:02.910 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:02.911 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:02.911 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:02.912 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.912 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:02.912 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.912 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.912 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:02.913 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.594 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:04.594 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:05.596 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:06.599 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:06.910 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:06.911 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:06.912 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:06.912 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:06.912 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:06.912 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:06.912 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:06.912 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:06.912 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.913 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:06.913 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.913 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.913 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:06.913 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.601 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:08.602 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:08.966 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:09.604 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:10.607 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:10.912 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:10.915 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:10.916 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:10.916 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:10.916 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:10.916 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:10.917 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:10.917 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:10.918 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.918 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:10.918 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.918 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.919 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:10.919 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.610 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:12.612 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:13.614 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:14.616 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:14.916 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:14.918 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:14.919 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:14.919 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:14.919 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:14.920 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:14.920 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:14.921 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:14.921 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.921 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:14.922 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.922 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.922 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:14.922 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.618 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:16.620 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:17.621 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:18.624 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:18.919 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:18.920 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:18.920 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:18.920 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:18.921 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:18.921 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:18.921 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:18.921 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:18.921 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.921 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:18.921 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.922 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.922 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:18.922 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.625 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:20.628 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:21.630 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:21.785 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:22.632 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:22.921 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:22.924 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:22.924 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:22.924 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:22.925 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:22.925 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:22.925 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:22.926 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:22.926 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.926 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:22.927 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.927 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.927 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:22.927 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.634 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:24.635 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:25.637 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:26.638 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:26.925 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:26.928 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:26.928 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:26.928 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:26.929 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:26.929 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:26.929 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:26.930 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:26.931 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.931 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:26.931 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.931 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.932 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:26.932 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.641 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:28.643 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:29.645 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:30.648 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:30.928 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:30.930 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:30.930 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:30.930 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:30.930 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:30.930 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:30.930 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:30.931 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:30.931 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.931 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:30.931 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.931 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.931 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:30.931 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.650 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:32.652 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:33.655 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:34.473 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:09:34.474 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.474 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:09:34.474 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.656 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:34.712 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:34.931 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:34.933 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:34.934 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:34.934 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:34.934 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:34.934 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:34.935 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:34.935 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:34.936 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.936 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:34.936 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.937 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.937 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:34.937 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.658 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:36.661 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:37.664 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:38.665 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:38.934 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:38.937 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:38.937 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:38.938 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:38.938 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:38.939 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:38.939 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:38.940 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:38.940 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.941 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:38.941 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.941 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.942 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:38.942 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.668 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:40.670 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:41.672 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:42.674 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:42.938 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:42.941 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:42.942 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:42.942 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:42.942 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:42.942 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:42.943 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:42.943 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:42.944 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.944 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:42.944 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.944 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.945 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:42.945 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.677 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:44.679 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:45.681 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:46.683 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:46.942 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:46.943 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:46.944 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:46.944 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:46.944 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:46.944 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:46.944 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:46.944 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:46.944 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.944 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:46.944 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.945 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.945 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:46.945 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.684 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:47.700 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:48.686 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:49.688 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:50.689 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:50.945 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:50.946 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:50.946 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:50.946 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:50.946 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:50.946 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:50.946 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:50.947 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:50.947 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.947 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:50.947 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.947 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.947 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:50.947 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.691 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:52.693 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:53.695 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:54.696 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:54.947 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:54.949 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:54.949 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:54.949 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:54.949 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:54.950 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:54.950 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:54.950 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:54.951 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.951 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:54.951 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.951 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.951 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:54.951 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.698 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:56.700 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:57.702 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:58.704 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:58.950 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:58.953 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:58.954 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:58.954 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:58.954 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:58.954 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:58.955 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:58.955 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:58.956 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.956 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:58.956 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.956 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.957 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:58.957 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.706 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:00.303 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:00.707 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:01.709 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:02.710 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:02.954 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:02.955 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:02.955 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:02.956 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:02.956 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:02.956 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:02.956 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:02.956 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:02.956 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.957 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:02.957 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.957 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.957 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:02.957 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.711 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:04.713 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:05.714 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:06.716 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:06.956 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:06.957 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:06.957 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:06.957 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.958 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:06.958 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.958 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:06.958 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:06.958 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.958 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:06.958 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.958 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.958 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:06.959 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.718 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:08.719 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:09.720 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:10.723 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:10.958 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:10.961 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:10.962 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:10.962 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.963 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:10.963 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.963 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:10.964 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:10.964 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.964 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:10.965 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.965 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.965 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:10.965 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.724 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:12.725 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:13.271 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:13.727 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:14.728 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:14.962 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:14.965 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:14.965 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:14.965 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.965 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:14.965 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.965 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:14.966 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:14.966 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.966 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:14.966 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.967 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.967 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:14.967 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.729 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:16.731 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:17.733 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:18.736 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:18.966 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:18.968 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:18.968 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:18.968 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.968 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:18.969 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.969 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:18.969 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:18.970 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.970 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:18.970 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.970 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.971 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:18.971 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.738 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:20.740 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:21.742 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:22.744 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:22.969 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:22.970 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:22.970 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:22.970 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.971 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:22.971 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.971 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:22.971 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:22.971 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.972 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:22.972 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.972 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.972 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:22.972 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.746 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:24.748 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:25.748 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:26.041 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:26.750 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:26.970 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:26.971 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:26.972 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:26.972 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.972 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:26.972 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.972 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:26.973 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:26.973 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.973 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:26.973 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.973 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.973 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:26.973 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.752 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:28.755 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:29.757 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:30.759 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:30.973 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:31.036 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:31.037 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:31.037 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:31.037 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:31.037 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:31.037 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:31.038 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:31.038 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.038 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:31.038 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.038 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.038 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:31.039 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.760 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:32.762 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:33.764 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:34.475 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:10:34.476 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.476 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:10:34.476 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.765 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:35.038 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:35.040 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:35.040 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:35.041 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:35.041 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:35.041 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:35.042 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:35.042 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:35.044 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.044 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:35.044 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.044 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.045 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:35.045 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.453 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) 09/01/2025 02:10:35.455 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) is either full or getting a new batch 09/01/2025 02:10:35.455 7f3921632cc0 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:10:35.456 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:10:35.456 7f3921632cc0 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) 09/01/2025 02:10:35.458 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:10:35.458 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8f %{\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n_?\x00\x00\x01\x99\x03\n_?\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:10:35.458 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8f %{\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n_?\x00\x00\x01\x99\x03\n_?\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:10:35.459 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8f %{\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n_?\x00\x00\x01\x99\x03\n_?\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:10:35.459 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 11 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8f %{\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n_?\x00\x00\x01\x99\x03\n_?\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:10:35.460 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 59881.611084, idle connection 179001.000000, request 29999.896288 09/01/2025 02:10:35.461 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.979506, metadata 59880.449463, idle connection 178999.000000, request 29998.747349 09/01/2025 02:10:35.469 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 11 09/01/2025 02:10:35.469 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:10:35.470 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 11 (10.43558120727539 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:10:35.471 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:10:35.471 7f392155b4a0 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) with base offset 0 log start offset 0. 09/01/2025 02:10:35.472 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.964008, metadata 59868.999756, idle connection 178988.000000, request inf 09/01/2025 02:10:35.473 7f3921632cc0 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:10:35.767 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:36.769 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:37.771 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:38.773 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:39.041 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:39.043 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:39.043 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:39.043 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:39.044 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:39.044 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:39.044 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:39.044 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:39.044 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.044 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:39.044 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.044 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.045 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:39.045 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.058 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:39.774 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:40.776 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:41.778 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:42.780 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:43.044 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:43.045 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:43.045 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:43.045 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:43.046 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:43.046 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:43.046 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:43.046 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:43.046 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.046 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:43.046 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.046 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.047 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:43.047 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.783 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:44.785 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:45.787 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:46.789 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:47.046 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:47.049 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:47.049 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:47.050 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:47.050 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:47.051 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:47.051 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:47.052 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:47.053 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.053 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:47.054 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.054 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.054 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:47.055 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.790 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:48.792 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:49.793 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:50.795 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:51.049 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:51.051 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:51.051 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:51.051 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:51.051 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:51.051 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:51.051 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:51.052 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:51.052 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.052 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:51.052 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.052 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.052 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:51.053 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.797 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:52.000 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:52.799 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:53.801 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:54.803 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:55.051 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:55.053 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:55.053 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:55.053 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:55.054 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:55.054 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:55.054 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:55.054 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:55.055 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.055 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:55.055 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.055 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.055 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:55.055 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.805 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:56.807 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:57.809 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:58.811 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:59.054 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:59.056 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:59.057 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:59.057 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:59.057 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:59.058 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:59.058 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:59.058 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:59.059 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.059 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:59.059 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.059 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.060 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:59.060 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.812 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:00.814 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:01.816 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:02.817 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:03.057 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:03.059 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:03.060 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:03.060 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:03.060 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:03.061 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:03.061 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:03.062 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:03.062 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.062 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:03.062 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.063 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.063 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:03.063 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.819 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:04.803 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:04.821 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:05.823 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:06.826 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:07.060 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:07.062 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:07.062 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:07.062 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:07.063 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:07.063 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:07.063 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:07.063 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:07.064 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.064 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:07.064 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.064 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.064 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:07.065 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.828 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:08.829 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:09.831 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:10.834 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:11.063 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:11.066 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:11.066 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:11.066 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:11.067 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:11.067 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:11.067 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:11.068 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:11.069 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.069 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:11.070 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.070 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.070 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:11.070 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.836 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:12.839 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:13.840 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:14.842 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:15.067 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:15.096 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:15.096 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:15.097 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:15.097 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:15.097 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:15.098 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:15.098 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:15.099 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.099 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:15.099 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.099 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.100 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:15.100 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.844 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:16.847 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:17.849 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:18.029 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:18.852 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:19.097 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:19.099 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:19.100 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:19.100 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:19.101 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:19.101 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:19.102 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:19.103 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:19.103 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.104 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:19.104 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.105 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.105 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:19.106 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.854 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:20.856 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:21.857 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:22.859 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:23.100 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:23.102 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:23.102 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:23.102 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:23.103 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:23.103 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:23.103 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:23.104 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:23.104 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.105 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:23.105 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.105 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.106 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:23.106 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.861 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:24.863 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:25.865 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:26.866 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:27.103 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:27.105 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:27.105 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:27.106 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:27.106 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:27.106 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:27.106 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:27.107 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:27.108 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.108 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:27.108 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.108 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.109 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:27.109 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.869 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:28.871 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:29.873 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:30.877 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:30.938 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:31.105 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:31.107 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:31.107 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:31.107 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:31.108 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:31.108 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:31.108 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:31.109 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:31.109 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.110 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:31.110 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.110 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.110 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:31.110 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.879 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:32.880 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:33.882 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:34.476 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:11:34.476 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.477 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:11:34.477 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.884 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:35.108 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:35.110 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:35.110 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:35.111 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:35.111 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:35.111 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:35.111 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:35.112 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:35.113 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.113 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:35.113 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.114 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.114 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:35.114 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.343 7f392155b4a0 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:11:35.343 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:11:35.343 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 12 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:11:35.343 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.416599, metadata 30000.000000, idle connection 119117.000000, request 29999.973297 09/01/2025 02:11:35.344 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993095, metadata 30000.000000, idle connection 119116.000000, request 29999.546528 09/01/2025 02:11:35.346 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 12 09/01/2025 02:11:35.346 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:11:35.347 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 12 (3.294706344604492 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.254', port=9092, rack=None)], cluster_id='J9jkOynITIuGnNhnlMXSzw', controller_id=1, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=5, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=10, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=20, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=25, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=26, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=11, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=4, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=16, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=17, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=23, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=28, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=13, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=22, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=8, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=2, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=29, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=19, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=14, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=1, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=7, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[])])]) 09/01/2025 02:11:35.348 7f392155b4a0 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 1, groups: 0) 09/01/2025 02:11:35.348 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992619, metadata 299999.858154, idle connection 119112.000000, request inf 09/01/2025 02:11:35.886 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:36.887 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:37.890 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:38.892 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:39.111 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:39.113 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:39.113 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:39.114 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:39.114 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:39.114 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:39.114 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:39.115 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:39.116 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.116 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:39.116 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.116 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.117 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:39.117 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.894 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:40.896 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:41.898 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:42.900 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:43.113 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:43.114 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:43.115 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:43.115 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:43.115 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:43.115 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:43.115 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:43.115 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:43.115 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.116 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:43.116 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.116 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.116 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:43.116 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.839 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:43.902 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:44.904 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:45.906 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:46.908 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:47.115 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:47.118 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:47.118 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:47.118 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:47.119 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:47.119 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:47.119 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:47.120 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:47.120 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.121 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:47.121 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.121 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.122 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:47.122 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.910 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:48.912 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:49.914 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:50.916 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:51.118 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:51.121 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:51.122 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:51.122 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:51.123 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:51.123 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:51.123 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:51.124 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:51.125 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.125 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:51.125 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.126 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.126 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:51.127 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.917 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:52.920 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:53.922 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:54.923 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:55.121 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:55.122 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:55.123 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:55.123 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:55.123 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:55.123 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:55.123 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:55.126 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:55.126 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.126 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:55.126 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.126 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.127 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:55.127 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.925 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:56.823 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:56.927 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:57.929 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:58.931 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:59.124 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:59.126 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:59.126 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:59.127 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:59.127 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:59.128 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:59.128 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:59.129 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:59.129 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.129 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:59.130 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.130 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.130 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:59.130 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.933 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:00.935 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:01.937 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:02.938 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:03.126 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:03.129 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:03.129 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:03.129 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:03.130 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:03.130 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:03.130 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:03.131 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:03.132 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.132 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:03.132 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.133 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.133 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:03.133 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.941 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:04.944 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:05.945 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:06.947 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:07.129 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:07.132 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:07.132 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:07.133 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:07.133 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:07.133 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:07.134 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:07.134 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:07.135 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.135 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:07.135 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.136 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.136 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:07.136 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.949 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:08.951 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:09.706 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:09.953 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:10.954 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:11.132 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:11.133 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:11.133 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:11.133 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:11.134 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:11.134 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:11.134 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:11.134 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:11.134 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.134 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:11.134 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.134 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.134 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:11.134 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.956 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:12.958 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:13.960 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:14.962 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:15.134 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:15.137 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:15.138 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:15.138 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:15.138 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:15.139 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:15.139 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:15.139 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:15.140 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.140 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:15.140 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.140 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.141 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:15.141 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.964 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:16.965 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:17.967 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:18.969 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:19.138 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:19.139 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:19.139 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:19.139 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:19.140 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:19.140 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:19.140 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:19.140 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:19.140 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.140 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:19.141 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.141 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.141 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:19.141 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.971 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:20.973 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:21.973 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:22.837 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:22.975 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:23.140 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:23.143 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:23.143 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:23.144 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:23.144 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:23.144 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:23.145 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:23.146 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:23.146 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.147 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:23.147 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.147 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.147 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:23.148 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.977 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:24.978 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:25.981 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:26.982 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:27.143 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:27.145 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:27.146 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:27.146 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:27.147 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:27.147 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:27.147 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:27.148 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:27.148 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.148 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:27.149 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.149 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.149 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:27.149 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.984 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:28.986 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:29.987 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:30.989 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:31.147 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:31.149 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:31.149 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:31.149 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:31.150 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:31.150 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:31.150 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:31.151 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:31.151 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.152 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:31.152 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.152 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.152 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:31.153 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.992 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:32.993 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:33.995 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:34.477 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:12:34.477 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.478 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:12:34.478 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.997 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:35.150 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:35.152 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:35.153 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:35.153 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:35.153 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:35.154 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:35.154 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:35.155 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:35.155 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.156 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:35.156 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.156 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.156 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:35.157 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.475 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=11) 09/01/2025 02:12:35.475 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=11) is either full or getting a new batch 09/01/2025 02:12:35.476 7f3921632cc0 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:12:35.476 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:12:35.476 7f3921632cc0 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=11) 09/01/2025 02:12:35.478 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:12:35.478 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02R\x84\xb7\x8d\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0c4\x13\x00\x00\x01\x99\x03\x0c4\x13\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:12:35.479 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02R\x84\xb7\x8d\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0c4\x13\x00\x00\x01\x99\x03\x0c4\x13\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:12:35.479 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02R\x84\xb7\x8d\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0c4\x13\x00\x00\x01\x99\x03\x0c4\x13\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:12:35.480 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 13 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02R\x84\xb7\x8d\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0c4\x13\x00\x00\x01\x99\x03\x0c4\x13\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:12:35.481 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 239867.224365, idle connection 58980.000000, request 29999.897003 09/01/2025 02:12:35.482 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978790, metadata 239866.015381, idle connection 58979.000000, request 29998.702288 09/01/2025 02:12:35.484 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 13 09/01/2025 02:12:35.485 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:12:35.485 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 13 (4.534482955932617 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:12:35.486 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:12:35.486 7f392155b4a0 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=11) with base offset 0 log start offset 0. 09/01/2025 02:12:35.486 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978790, metadata 239861.256348, idle connection 58974.000000, request inf 09/01/2025 02:12:35.487 7f3921632cc0 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:12:35.592 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:35.999 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:37.001 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:38.001 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:39.003 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:39.153 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:39.155 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:39.156 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:39.156 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:39.156 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:39.157 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:39.157 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:39.158 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:39.158 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.158 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:39.159 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.159 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.159 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:39.159 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:40.005 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:41.007 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:42.010 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:43.012 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:43.156 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:43.158 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:43.158 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:43.159 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:43.159 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:43.159 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:43.160 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:43.160 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:43.161 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.161 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:43.162 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.162 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.162 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:43.162 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.015 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:45.016 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:46.018 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:47.021 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:47.159 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:47.161 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:47.162 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:47.162 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:47.163 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:47.163 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:47.163 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:47.164 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:47.166 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.167 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:47.167 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.167 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.167 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:47.168 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:48.022 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:48.623 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:49.025 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:50.027 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:51.029 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:51.162 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:51.164 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:51.165 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:51.165 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:51.165 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:51.166 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:51.166 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:51.167 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:51.167 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.167 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:51.167 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.168 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.168 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:51.168 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.032 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:53.034 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:54.035 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:55.037 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:55.165 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:55.168 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:55.169 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:55.169 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:55.169 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:55.169 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:55.170 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:55.170 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:55.171 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.171 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:55.171 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.172 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.172 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:55.172 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:56.039 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:57.040 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:58.042 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:59.044 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:59.168 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:59.170 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:59.171 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:59.171 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:59.172 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:59.172 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:59.172 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:59.173 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:59.173 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.173 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:59.174 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.174 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.174 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:59.174 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.046 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:01.048 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:01.571 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:02.050 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:03.052 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:03.171 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:03.174 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:03.174 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:03.174 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:03.175 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:03.175 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:03.175 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:03.176 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:03.176 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.177 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:03.177 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.177 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.177 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:03.178 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.054 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:05.056 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:06.057 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:07.059 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:07.174 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:07.176 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:07.177 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:07.177 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:07.178 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:07.178 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:07.178 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:07.179 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:07.180 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.181 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:07.181 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.181 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.182 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:07.182 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.061 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:09.063 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:10.065 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:11.068 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:11.177 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:11.181 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:11.181 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:11.182 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:11.182 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:11.183 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:11.183 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:11.184 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:11.185 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.185 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:11.186 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.186 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.187 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:11.187 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.070 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:13.071 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:14.073 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:14.425 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:15.075 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:15.181 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:15.184 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:15.184 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:15.185 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:15.185 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:15.185 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:15.185 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:15.186 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:15.187 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.187 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:15.187 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.187 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.188 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:15.188 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.077 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:17.080 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:18.082 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:19.084 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:19.184 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:19.186 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:19.187 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:19.187 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:19.187 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:19.187 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:19.188 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:19.188 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:19.189 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.189 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:19.189 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.189 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.190 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:19.190 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.086 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:21.088 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:22.090 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:23.091 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:23.187 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:23.189 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:23.189 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:23.189 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:23.190 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:23.190 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:23.190 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:23.191 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:23.191 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.192 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:23.192 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.192 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.192 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:23.193 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.093 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:25.095 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:26.097 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:27.099 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:27.190 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:27.192 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:27.193 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:27.193 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:27.194 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:27.194 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:27.194 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:27.195 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:27.196 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.196 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:27.196 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.197 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.197 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:27.197 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.783 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:28.101 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:29.103 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:30.105 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:31.107 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:31.194 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:31.196 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:31.196 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:31.196 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:31.197 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:31.197 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:31.197 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:31.198 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:31.198 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.199 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:31.199 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.199 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.199 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:31.200 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.109 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:33.110 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:34.113 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:34.463 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.972115, metadata 180884.717773, idle connection 481021.000000, request inf 09/01/2025 02:13:34.478 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:13:34.479 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.479 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:13:34.480 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.115 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:35.196 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:35.197 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:35.197 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:35.197 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:35.198 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:35.198 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:35.198 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:35.198 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:35.198 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.198 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:35.198 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.198 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.199 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:35.199 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:36.116 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:37.119 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:38.121 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:39.122 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:39.198 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:39.200 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:39.200 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:39.200 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:39.201 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:39.201 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:39.201 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:39.202 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:39.202 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.203 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:39.203 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.203 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.204 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:39.204 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.124 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:40.657 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:41.126 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:42.128 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:43.129 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:43.201 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:43.203 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:43.204 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:43.204 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:43.204 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:43.204 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:43.205 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:43.205 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:43.206 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.206 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:43.206 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.206 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.207 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:43.207 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.132 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:45.134 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:46.136 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:47.138 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:47.204 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:47.206 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:47.206 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:47.207 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:47.207 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:47.207 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:47.207 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:47.208 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:47.209 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.209 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:47.209 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.210 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.210 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:47.210 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:48.140 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:49.142 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:50.145 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:51.147 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:51.206 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:51.208 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:51.209 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:51.209 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:51.209 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:51.210 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:51.210 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:51.211 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:51.211 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.211 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:51.211 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.212 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.212 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:51.212 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.149 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:53.151 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:53.426 7f39215c82c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:54.153 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:55.155 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:55.209 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:55.211 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:55.212 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:55.212 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:55.212 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:55.213 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:55.213 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:55.214 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:55.214 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.214 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:55.215 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.215 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.215 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:55.215 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:56.157 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:57.158 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:58.160 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:59.161 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:59.212 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:59.214 7f39215c82c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:59.214 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:59.214 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:59.214 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:59.214 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:59.214 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:59.215 7f39215c82c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:59.215 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.215 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:59.215 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.215 7f39215c82c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.215 7f39215c82c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:59.215 7f39215c82c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:00.163 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:01.165 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:02.168 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:03.170 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:03.215 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:03.216 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:03.216 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:03.216 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:03.216 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:03.216 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:03.217 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:03.217 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:03.217 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.217 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:03.217 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.217 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.218 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:03.218 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.172 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:05.174 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:06.126 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:06.176 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:07.178 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:07.216 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:07.217 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:07.218 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:07.218 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:07.218 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:07.218 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:07.218 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:07.218 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:07.218 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.219 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:07.219 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.219 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.219 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:07.219 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.181 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:09.184 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:10.185 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:11.188 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:11.218 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:11.220 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:11.221 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:11.221 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:11.222 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:11.222 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:11.222 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:11.223 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:11.223 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.224 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:11.224 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.224 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.224 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:11.225 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:12.191 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:13.193 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:14.193 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:15.195 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:15.222 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:15.224 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:15.224 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:15.224 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:15.225 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:15.225 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:15.225 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:15.226 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:15.226 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.227 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:15.227 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.227 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.227 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:15.228 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.197 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:17.200 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:18.216 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:19.006 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:19.219 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:19.225 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:19.227 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:19.227 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:19.228 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:19.228 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:19.228 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:19.228 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:19.229 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:19.230 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.230 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:19.231 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.231 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.231 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:19.232 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:20.221 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:21.222 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:22.224 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:23.226 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:23.228 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:23.230 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:23.230 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:23.231 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:23.231 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:23.231 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:23.231 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:23.232 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:23.233 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.233 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:23.233 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.233 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.234 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:23.234 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:24.229 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:25.231 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:26.233 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:27.231 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:27.233 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:27.234 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:27.234 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:27.234 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:27.235 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:27.235 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:27.236 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:27.236 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.236 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:27.236 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.237 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.237 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:27.237 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.238 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:28.239 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:29.240 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:30.241 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:31.235 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:31.238 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:31.238 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:31.238 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:31.239 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:31.239 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:31.239 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:31.240 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:31.240 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.241 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:31.241 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.241 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.241 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:31.242 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.243 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:32.064 7f392155b540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:32.244 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:33.246 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:34.248 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:34.479 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:14:34.480 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.480 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:14:34.480 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.239 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:35.241 7f392155b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:35.242 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:35.242 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:35.243 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:35.243 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:35.243 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:35.244 7f392155b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:35.245 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.245 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:35.245 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.246 7f392155b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.246 7f392155b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:35.246 7f392155b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.250 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:35.488 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=10) 09/01/2025 02:14:35.488 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=10) is either full or getting a new batch 09/01/2025 02:14:35.489 7f3921632cc0 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:14:35.489 7f3921632cc0 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:14:35.489 7f3921632cc0 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=10) 09/01/2025 02:14:35.489 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:14:35.489 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=10, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xd8\xfdTF\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0e\x08\xe0\x00\x00\x01\x99\x03\x0e\x08\xe0\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:14:35.489 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=10, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xd8\xfdTF\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0e\x08\xe0\x00\x00\x01\x99\x03\x0e\x08\xe0\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:14:35.490 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=10, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xd8\xfdTF\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0e\x08\xe0\x00\x00\x01\x99\x03\x0e\x08\xe0\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:14:35.490 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Request 14 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=10, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xd8\xfdTF\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0e\x08\xe0\x00\x00\x01\x99\x03\x0e\x08\xe0\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:14:35.490 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 119857.852295, idle connection 419994.000000, request 29999.964237 09/01/2025 02:14:35.490 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994287, metadata 119857.423096, idle connection 419993.000000, request 29999.541521 09/01/2025 02:14:35.496 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Received correlation id: 14 09/01/2025 02:14:35.497 7f392155b4a0 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:14:35.497 7f392155b4a0 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.254', 9092)]>: Response 14 (7.462978363037109 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=10, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:14:35.498 7f392155b4a0 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=10, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:14:35.498 7f392155b4a0 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=10) with base offset 0 log start offset 0. 09/01/2025 02:14:35.499 7f392155b4a0 [kafka.client] [DEBUG]: Timeouts: user 9999999989.969969, metadata 119848.774170, idle connection 419985.000000, request inf 09/01/2025 02:14:35.499 7f3921632cc0 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:14:36.251 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:37.253 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:38.255 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:39.242 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:39.245 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:39.245 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:39.246 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:39.246 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:39.246 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:39.247 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:39.248 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:39.248 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.249 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:39.249 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.249 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.249 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:39.250 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.257 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:40.259 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:41.261 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:42.262 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:43.246 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:43.248 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:43.248 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:43.248 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:43.249 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:43.249 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:43.250 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:43.251 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:43.251 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.251 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:43.252 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.252 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.252 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:43.252 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.264 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:44.266 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:44.937 7f392155b5e0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:45.268 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:46.280 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:47.249 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:47.252 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:47.252 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:47.253 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:47.253 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:47.253 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:47.254 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:47.254 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:47.255 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.255 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:47.255 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.256 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.256 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:47.256 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.283 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:48.286 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:49.288 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:50.290 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:51.252 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:51.254 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:51.254 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:51.254 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:51.254 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:51.255 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:51.255 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:51.255 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:51.255 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.255 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:51.256 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.256 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.256 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:51.256 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.292 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:52.293 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:53.295 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:54.297 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:55.255 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:55.257 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:55.257 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:55.257 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:55.257 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:55.257 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:55.257 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:55.258 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:55.258 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.258 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:55.258 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.258 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.258 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:55.258 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.299 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:56.300 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:57.302 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:57.885 7f392155b5e0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:58.303 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:59.258 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:59.261 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:59.261 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:59.262 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:59.262 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:59.262 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:59.262 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:59.263 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:59.263 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.264 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:59.264 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.264 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.264 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:59.265 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.305 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:00.307 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:01.308 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:02.309 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:03.261 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:03.262 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.263 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:03.264 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.311 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:04.312 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:05.314 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:06.316 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:07.264 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:07.267 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:07.267 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:07.267 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:07.268 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:07.268 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:07.269 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:07.270 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:07.271 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:07.271 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:07.272 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:07.272 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:07.272 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:07.273 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:07.317 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:08.319 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:09.321 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:10.323 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:10.948 7f392155b5e0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:11.268 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:11.270 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:11.270 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:11.270 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:11.271 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:11.271 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:11.271 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:11.272 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:11.272 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:11.273 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:11.273 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:11.273 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:11.273 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:11.274 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:11.324 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:12.332 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:13.334 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:14.335 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:15.270 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:15.273 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:15.273 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:15.273 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:15.274 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:15.274 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:15.274 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:15.275 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:15.277 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:15.277 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:15.278 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:15.278 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:15.278 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:15.278 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:15.338 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:16.339 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:17.341 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:18.342 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:19.273 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:19.276 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:19.276 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:19.276 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:19.277 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:19.277 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:19.277 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:19.278 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:19.278 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:19.279 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:19.279 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:19.279 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:19.279 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:19.279 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:19.344 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:20.345 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:21.347 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:22.348 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:23.277 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:23.280 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:23.280 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:23.281 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:23.281 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:23.281 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:23.281 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:23.282 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:23.283 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:23.283 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:23.284 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:23.284 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:23.284 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:23.284 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:23.350 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:23.793 7f392155b5e0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:24.351 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:25.354 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:26.354 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:27.281 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:27.283 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:27.283 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:27.284 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:27.284 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:27.284 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:27.284 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:27.285 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:27.286 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:27.286 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:27.286 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:27.286 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:27.287 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:27.287 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:27.357 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:28.358 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:29.360 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:30.362 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:31.285 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:31.287 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:31.287 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:31.287 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:31.288 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:31.288 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:31.288 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:31.289 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:31.289 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:31.290 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:31.290 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:31.290 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:31.290 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:31.291 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:31.363 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:32.364 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:33.366 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:34.367 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:34.480 7f3921632540 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:15:34.480 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:34.481 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:15:34.481 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:35.288 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:35.289 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:35.289 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:35.289 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:35.290 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:35.290 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:35.290 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:35.290 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:35.290 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:35.290 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:35.291 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:35.291 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:35.291 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:35.291 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:35.369 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:36.372 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:37.053 7f392155b5e0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:37.373 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:38.376 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:39.290 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:39.293 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:39.293 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:39.293 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:39.294 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:39.294 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:39.294 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:39.295 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:39.295 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:39.296 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:39.296 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:39.296 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:39.297 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:39.297 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:39.377 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:39.798 7f392155b5e0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:40.380 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:41.381 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:42.384 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:42.704 7f392155b5e0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:43.293 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:43.296 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:43.296 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:43.296 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:43.297 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:43.297 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:43.297 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:43.302 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:43.302 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:43.303 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:43.303 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:43.303 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:43.303 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:43.303 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:43.385 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:44.387 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:45.388 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:46.391 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:47.297 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:47.300 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:47.301 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:47.301 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:47.302 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:47.302 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:47.302 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:47.303 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:47.303 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:47.303 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:47.304 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:47.304 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:47.304 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:47.305 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:47.393 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:48.396 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:49.398 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:50.400 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:51.302 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:51.304 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:51.304 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:51.304 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:51.304 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:51.305 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:51.305 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:51.305 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:51.305 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:51.305 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:51.306 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:51.306 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:51.306 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:51.306 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:51.402 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:52.404 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:53.405 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:54.407 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:55.305 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:55.306 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:55.306 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:55.307 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:55.307 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:55.307 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:55.307 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:55.307 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:55.308 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:55.308 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:55.308 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:55.308 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:55.308 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:55.308 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:55.408 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:56.409 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:57.412 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:58.413 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:59.308 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:59.310 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:59.310 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:59.311 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:59.311 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:59.311 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:59.312 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:59.312 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:59.313 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:59.313 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:59.313 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:59.314 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:59.314 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:59.314 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:59.417 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:00.419 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:01.421 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:02.424 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:03.312 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:03.314 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:03.314 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:03.315 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:03.315 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:03.315 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:03.316 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:03.316 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:03.317 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:03.317 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:03.317 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:03.318 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:03.318 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:03.318 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:03.426 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:04.427 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:05.429 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:06.431 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:07.315 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:07.318 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:07.318 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:07.319 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:07.319 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:07.319 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:07.320 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:07.320 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:07.321 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:07.321 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:07.321 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:07.322 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:07.322 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:07.322 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:07.432 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:08.433 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:09.436 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:10.438 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:11.320 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:11.323 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:11.323 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:11.323 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:11.324 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:11.324 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:11.325 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:11.326 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:11.326 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:11.327 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:11.327 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:11.327 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:11.328 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:11.328 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:11.440 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:12.443 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:13.445 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:14.447 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:15.323 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:15.325 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:15.326 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:15.326 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:15.327 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:15.327 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:15.327 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:15.328 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:15.329 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:15.329 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:15.330 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:15.330 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:15.330 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:15.330 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:15.449 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:16.535 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:17.079 7f392155b5e0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:16:17.537 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:18.539 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:19.327 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:19.331 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:19.331 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:19.331 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:19.332 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:19.332 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:19.332 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:19.333 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:19.333 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:19.334 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:19.334 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:19.334 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:19.334 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:19.335 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:19.540 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:20.542 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:21.544 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:22.545 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:23.331 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:23.333 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:23.333 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:23.333 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:23.334 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:23.334 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:23.334 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:23.334 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:23.334 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:23.335 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:23.335 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:23.335 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:23.335 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:23.335 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:23.547 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:24.549 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:25.551 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:26.553 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:27.333 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:27.335 7f392155b5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:27.335 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:27.335 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:27.335 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:27.335 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:27.335 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:27.336 7f392155b5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:27.336 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:27.336 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.254:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:27.336 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:27.336 7f392155b5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:27.336 7f392155b5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3992-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691974454689 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:27.336 7f392155b5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:27.555 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:28.557 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:29.559 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:30.561 7f3921632a40 [contrail-alarm-gen] [INFO]: UVE Done