09/01/2025 01:51:48.866 7fc2b16c80c0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:51:48.866 7fc2b16c80c0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:51:48.866 7fc2b16c80c0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:51:48.873 7fc2b16c80c0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/01/2025 01:51:48.873 7fc2ad7b4d60 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/01/2025 01:51:48.873 7fc2ad7b4d60 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/01/2025 01:51:48.873 7fc2ad7d8c20 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:51:48.874 7fc2ad7d8c20 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:51:48.874 7fc2ad7d8c20 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:51:48.874 7fc2ad7d8c20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:51:48.874 7fc2ad7d8c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.874 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:51:48.874 7fc2ad7d8c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.874 7fc2ad7d8c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.874 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/01/2025 01:51:48.875 7fc2ad7d8c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.875 7fc2ad7d8c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.875 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.33:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:51:48.875 7fc2ad7d8c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.875 7fc2ad7d8c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.875 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.33:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:51:48.875 7fc2ad7d8c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.876 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:51:48.876 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/01/2025 01:51:48.877 7fc2ad780d60 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/01/2025 01:51:48.877 7fc2ad71bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.877 7fc2ad71bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:51:48.877 7fc2ad71bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.877 7fc2ad71bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.878 7fc2ad71bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.33:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:51:48.878 7fc2ad71bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.935 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.33:5673//'] 09/01/2025 01:51:48.942 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:51:48.943 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.943 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.33:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:51:48.943 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.943 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.943 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.33:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:51:48.943 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.944 7fc2ad72e720 [kazoo.client] [INFO]: Connecting to 10.0.0.33(10.0.0.33):2181, use_ssl: False 09/01/2025 01:51:48.945 7fc2ad72e720 [kazoo.client] [DEBUG]: Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=60000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) 09/01/2025 01:51:48.945 7fc2ad72e0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.945 7fc2ad72e0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:51:48.946 7fc2ad72e0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.946 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:51:48.947 7fc2ad72e0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.947 7fc2ad72e0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:51:48.947 7fc2ad72e0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.953 7fc2ad72e720 [kazoo.client] [INFO]: Zookeeper connection established, state: CONNECTED 09/01/2025 01:51:48.953 7fc2ad72e720 [contrail-alarm-gen] [INFO]: Analytics Discovery listen CONNECTED 09/01/2025 01:51:48.953 7fc2ad72e720 [contrail-alarm-gen] [INFO]: Analytics Discovery to publish None 09/01/2025 01:51:48.953 7fc2ad680e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.954 7fc2ad680e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper re-established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:51:48.954 7fc2ad680e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.954 7fc2ad680e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.954 7fc2ad680e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:51:48.954 7fc2ad680e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.954 7fc2ad72e720 [kazoo.client] [DEBUG]: Sending request(xid=1): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:51:48.955 7fc2ad72e720 [kazoo.client] [DEBUG]: Received response(xid=1): ZnodeStat(czxid=213, mzxid=213, ctime=1756691495974, mtime=1756691495974, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=213) 09/01/2025 01:51:48.955 7fc2ad72e720 [kazoo.client] [DEBUG]: Sending request(xid=2): GetChildren(path='/analytics-discovery-/AlarmGenerator', watcher=>) 09/01/2025 01:51:48.956 7fc2ad72e720 [kazoo.client] [DEBUG]: Received response(xid=2): [] 09/01/2025 01:51:48.956 7fc2ad780d60 [contrail-alarm-gen] [INFO]: Analytics Discovery Watcher AlarmGenerator Children [] 09/01/2025 01:51:48.956 7fc2ad72e720 [kazoo.client] [DEBUG]: Sending request(xid=3): Exists(path='/analytics-discovery-/Collector', watcher=None) 09/01/2025 01:51:48.957 7fc2ad72e720 [kazoo.client] [DEBUG]: Received response(xid=3): ZnodeStat(czxid=214, mzxid=214, ctime=1756691495978, mtime=1756691495978, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=214) 09/01/2025 01:51:48.957 7fc2ad72e720 [kazoo.client] [DEBUG]: Sending request(xid=4): GetChildren(path='/analytics-discovery-/Collector', watcher=>) 09/01/2025 01:51:48.958 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:51:48.958 7fc2ad680860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.958 7fc2ad680860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:51:48.958 7fc2ad680860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.959 7fc2ad72e720 [kazoo.client] [DEBUG]: Received response(xid=4): [] 09/01/2025 01:51:48.959 7fc2ad780d60 [contrail-alarm-gen] [INFO]: Analytics Discovery AlarmGenerator reconnect 09/01/2025 01:51:48.959 7fc2ad780d60 [contrail-alarm-gen] [INFO]: ensure /analytics-discovery-/AlarmGenerator 09/01/2025 01:51:48.959 7fc2ad780d60 [contrail-alarm-gen] [INFO]: zk state CONNECTED (CONNECTED) 09/01/2025 01:51:48.959 7fc2ad72e720 [kazoo.client] [DEBUG]: Sending request(xid=5): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:51:48.960 7fc2ad72e720 [kazoo.client] [DEBUG]: Received response(xid=5): ZnodeStat(czxid=213, mzxid=213, ctime=1756691495974, mtime=1756691495974, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=213) 09/01/2025 01:51:48.961 7fc2ad780d60 [contrail-alarm-gen] [INFO]: check for /analytics-discovery-/AlarmGenerator/cn-jenkins-deploy-platform-ansible-os-3994-1.-0 09/01/2025 01:51:48.961 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:51:48.961 7fc2ad72e720 [kazoo.client] [DEBUG]: Sending request(xid=6): Exists(path='/analytics-discovery-/AlarmGenerator/cn-jenkins-deploy-platform-ansible-os-3994-1.-0', watcher=None) 09/01/2025 01:51:48.961 7fc2ad6809a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.961 7fc2ad6809a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:51:48.961 7fc2ad6809a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.962 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/01/2025 01:51:48.962 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/01/2025 01:51:48.962 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:51:48.962 7fc2ad6809a0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:51:48.963 7fc2ad62fe00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.963 7fc2ad62fe00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:51:48.963 7fc2ad62fe00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.963 7fc2ad62fe00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.963 7fc2ad62fe00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:51:48.963 7fc2ad62fe00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.966 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:51:48.967 7fc2ad72e720 [kazoo.client] [DEBUG]: Sending request(xid=7): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:51:48.969 7fc2ad6809a0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:51:48.969 7fc2ad644180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.969 7fc2ad644180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:51:48.969 7fc2ad644180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.970 7fc2ad72e720 [kazoo.client] [DEBUG]: Received response(xid=7): ZnodeStat(czxid=213, mzxid=213, ctime=1756691495974, mtime=1756691495974, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=213) 09/01/2025 01:51:48.970 7fc2ad72e720 [kazoo.client] [DEBUG]: Sending request(xid=8): GetChildren(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:51:48.971 7fc2ad72e720 [kazoo.client] [DEBUG]: Received response(xid=8): [] 09/01/2025 01:51:48.971 7fc2ad780d60 [contrail-alarm-gen] [INFO]: Discovery AG callback : dict_values([]) 09/01/2025 01:51:48.984 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:51:48.986 7fc2ad6585e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.986 7fc2ad6585e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:51:48.986 7fc2ad6585e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.990 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:51:48.990 7fc2ad6584a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.990 7fc2ad6584a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:51:48.990 7fc2ad6584a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:48.993 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:51:49.002 7fc2ad7d8c20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:51:49.003 7fc2ad615680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:49.003 7fc2ad615680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:51:49.003 7fc2ad615680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:49.021 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config config_type = global_system_config operation = UPDATE config = { uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"fq_name : ["default-global-system-config"]autonomous_system : 64512enable_4byte_as : falseconfig_version : "1.0"ibgp_auto_mesh : truemac_aging_time : 300igmp_enable : falseenable_security_policy_draft : falseid_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17161074955751669897, "uuid_lslong": 9918450551028808763}, "enable": true, "created": "2025-09-01T01:49:37.161933", "last_modified": "2025-09-01T01:50:32.791194", "description": null, "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3994-1"], "attr": null, "uuid": "530bf36f-6219-4a8c-9c41-50ea3eeac60b"}] } 09/01/2025 01:51:49.022 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-address-mismatch-compute config_type = alarm operation = UPDATE config = { uuid : "6b8fb5c3-972a-4d39-b05a-87770f85cf7d"fq_name : ["default-global-system-config", "system-defined-address-mismatch-compute"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_ip_address", "operand2": {"uve_attribute": "VrouterAgent.control_ip", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7750613335387295033, "uuid_lslong": 12707618244010561405}, "enable": true, "created": "2025-09-01T01:49:47.903592", "last_modified": "2025-09-01T01:49:47.903592", "description": "Compute Node IP Address mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-address-mismatch-compute" } 09/01/2025 01:51:49.022 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-address-mismatch-control config_type = alarm operation = UPDATE config = { uuid : "2e16f078-191b-4772-b34d-a9ef21720df1"fq_name : ["default-global-system-config", "system-defined-address-mismatch-control"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "not in", "operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3321106173840279410, "uuid_lslong": 12920169750519221745}, "enable": true, "created": "2025-09-01T01:49:47.963450", "last_modified": "2025-09-01T01:49:47.963450", "description": "Control Node IP Address mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-address-mismatch-control" } 09/01/2025 01:51:49.022 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-bgp-connectivity config_type = alarm operation = UPDATE config = { uuid : "f3d1eec5-b7ec-47e0-b792-da523395ea66"fq_name : ["default-global-system-config", "system-defined-bgp-connectivity"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "BgpRouterState.num_up_bgp_peer", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "BgpRouterState.num_up_bgp_peer", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17569086154310698976, "uuid_lslong": 13227875102128204390}, "enable": true, "created": "2025-09-01T01:49:48.028798", "last_modified": "2025-09-01T01:49:48.028798", "description": "BGP peer mismatch. Not enough BGP peers are up.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-bgp-connectivity" } 09/01/2025 01:51:49.023 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-bottle-request-size-limit config_type = alarm operation = UPDATE config = { uuid : "62b6c9b7-7f85-45c5-9605-4fc814be8c44"fq_name : ["default-global-system-config", "system-defined-bottle-request-size-limit"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["config-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "UveConfigReq.err_info.*.", "operand2": {"uve_attribute": null, "json_value": "True"}, "variables": []}]}]}id_perms : {"permissions": null, "uuid": {"uuid_mslong": 7113094451433915845, "uuid_lslong": 10810134201332829252}, "enable": null, "created": "2025-09-01T01:49:37.699242", "last_modified": "2025-09-01T01:49:37.699242", "description": "Bottle request size limit exceeded.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []} } 09/01/2025 01:51:49.023 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-conf-elements-inconsistency config_type = alarm operation = UPDATE config = { uuid : "a2ab29ba-839b-423b-b498-4783ac86ef13"fq_name : ["default-global-system-config", "system-defined-conf-elements-inconsistency"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "size==", "operand1": "ContrailConfig.elements", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11721508336210494011, "uuid_lslong": 13013229754147860243}, "enable": true, "created": "2025-09-01T01:49:47.885625", "last_modified": "2025-09-01T01:49:47.885625", "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-conf-elements-inconsistency" } 09/01/2025 01:51:49.023 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-conf-incorrect config_type = alarm operation = UPDATE config = { uuid : "b7fd6b31-fd34-4157-8299-d51ae9a289a2"fq_name : ["default-global-system-config", "system-defined-conf-incorrect"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "ContrailConfig", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 13257870740494238039, "uuid_lslong": 9410787207932971426}, "enable": true, "created": "2025-09-01T01:49:48.092010", "last_modified": "2025-09-01T01:49:48.092010", "description": "ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-conf-incorrect" } 09/01/2025 01:51:49.024 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-core-files config_type = alarm operation = UPDATE config = { uuid : "401f889b-ec36-452d-ae23-cf862179a454"fq_name : ["default-global-system-config", "system-defined-core-files"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "NodeStatus.all_core_file_list", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "NodeStatus.all_core_file_list", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4620561945969706285, "uuid_lslong": 12548101161778521172}, "enable": true, "created": "2025-09-01T01:49:48.945624", "last_modified": "2025-09-01T01:49:48.945624", "description": "A core file has been generated on the node.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-core-files" } 09/01/2025 01:51:49.024 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-disk-usage-critical config_type = alarm operation = UPDATE config = { uuid : "4b8ec3f2-138f-4348-a82d-9c8149982a5d"fq_name : ["default-global-system-config", "system-defined-disk-usage-critical"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": ">", "operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operand2": {"uve_attribute": null, "json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 5444504444015166280, "uuid_lslong": 12118514251423296093}, "enable": true, "created": "2025-09-01T01:49:48.215147", "last_modified": "2025-09-01T01:49:48.215147", "description": "Disk usage crosses critical threshold limit.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-disk-usage-critical" } 09/01/2025 01:51:49.024 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-disk-usage-high config_type = alarm operation = UPDATE config = { uuid : "aa0b7d26-eb68-4216-8fad-93f9b3abc95f"fq_name : ["default-global-system-config", "system-defined-disk-usage-high"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "range", "operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operand2": {"uve_attribute": null, "json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12253024817303274006, "uuid_lslong": 10353093819065157983}, "enable": true, "created": "2025-09-01T01:49:48.152864", "last_modified": "2025-09-01T01:49:48.152864", "description": "Disk usage crosses high threshold limit.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-disk-usage-high" } 09/01/2025 01:51:49.024 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-node-status config_type = alarm operation = UPDATE config = { uuid : "61d0d462-36f4-43a8-8f3f-ebaacb47800f"fq_name : ["default-global-system-config", "system-defined-node-status"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7048366935128687528, "uuid_lslong": 10322227989743894543}, "enable": true, "created": "2025-09-01T01:49:48.277814", "last_modified": "2025-09-01T01:49:48.277814", "description": "Node Failure. NodeStatus UVE not present.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-node-status" } 09/01/2025 01:51:49.025 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-package-version-mismatch config_type = alarm operation = UPDATE config = { uuid : "61dea773-4c98-4929-9c76-e64c56e4d7db"fq_name : ["default-global-system-config", "system-defined-package-version-mismatch"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "NodeStatus.running_package_version", "operand2": {"uve_attribute": "NodeStatus.installed_package_version", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7052258180156901673, "uuid_lslong": 11274451932718356443}, "enable": true, "created": "2025-09-01T01:49:49.075787", "last_modified": "2025-09-01T01:49:49.075787", "description": "There is a mismatch between installed and running package version.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-package-version-mismatch" } 09/01/2025 01:51:49.025 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-partial-sysinfo config_type = alarm operation = UPDATE config = { uuid : "8345308e-fd09-4f0a-ac76-2ab52815fb8d"fq_name : ["default-global-system-config", "system-defined-partial-sysinfo"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.build_info", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9459019983050329866, "uuid_lslong": 12427167179325438861}, "enable": true, "created": "2025-09-01T01:49:48.341163", "last_modified": "2025-09-01T01:49:48.341163", "description": "System Info Incomplete.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-partial-sysinfo" } 09/01/2025 01:51:49.025 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks config_type = alarm operation = UPDATE config = { uuid : "0a33dd78-595f-4f6f-9e75-a25c30cd3685"fq_name : ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["database-nodeconfig-database-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": ">=", "operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"uve_attribute": null, "json_value": "300"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 735174673156755311, "uuid_lslong": 11418210947107206789}, "enable": true, "created": "2025-09-01T01:49:49.010454", "last_modified": "2025-09-01T01:49:49.010454", "description": "Pending compaction tasks in cassandra crossed the configured threshold.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-pending-cassandra-compaction-tasks" } 09/01/2025 01:51:49.025 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-process-connectivity config_type = alarm operation = UPDATE config = { uuid : "0affe545-29d0-47d8-a484-7adeadb5bfa4"fq_name : ["default-global-system-config", "system-defined-process-connectivity"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.process_status", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "NodeStatus.process_status.state", "operand2": {"uve_attribute": null, "json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 792604144657516504, "uuid_lslong": 11854735215961685924}, "enable": true, "created": "2025-09-01T01:49:48.404919", "last_modified": "2025-09-01T01:49:48.404919", "description": "Process(es) reporting as non-functional.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-process-connectivity" } 09/01/2025 01:51:49.026 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-process-status config_type = alarm operation = UPDATE config = { uuid : "e3e2b832-e521-4faa-b9fa-84983311c9f0"fq_name : ["default-global-system-config", "system-defined-process-status"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.process_info", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "NodeStatus.process_info.process_state", "operand2": {"uve_attribute": null, "json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16420889720078290858, "uuid_lslong": 13401169430421031408}, "enable": true, "created": "2025-09-01T01:49:48.467472", "last_modified": "2025-09-01T01:49:48.467472", "description": "Process Failure.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-process-status" } 09/01/2025 01:51:49.026 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-prouter-connectivity config_type = alarm operation = UPDATE config = { uuid : "ee19be47-058b-4977-90da-50e348b9ee16"fq_name : ["default-global-system-config", "system-defined-prouter-connectivity"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["prouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.connected_agent_list", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17156953467689585015, "uuid_lslong": 10437744023492488726}, "enable": true, "created": "2025-09-01T01:49:48.532111", "last_modified": "2025-09-01T01:49:48.532111", "description": "Prouter connectivity to controlling tor agent does not exist we look for non-empty value for connected_agent_list", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-prouter-connectivity" } 09/01/2025 01:51:49.026 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-prouter-tsn-connectivity config_type = alarm operation = UPDATE config = { uuid : "3376205c-3acb-4b74-88f4-8448b13c4687"fq_name : ["default-global-system-config", "system-defined-prouter-tsn-connectivity"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["prouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.tsn_agent_list", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}, {"operation": "!=", "operand1": "ProuterData.gateway_mode", "operand2": {"uve_attribute": null, "json_value": "\"SERVER\""}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.tsn_agent_list", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": []}, {"operation": "==", "operand1": "ProuterData.gateway_mode", "operand2": {"uve_attribute": null, "json_value": "\"SERVER\""}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3708186923681663860, "uuid_lslong": 9868658131221628551}, "enable": true, "created": "2025-09-01T01:49:48.592015", "last_modified": "2025-09-01T01:49:48.592015", "description": "Prouter connectivity to controlling tsn agent does not exist we look for non-empty value for tsn_agent_list", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-prouter-tsn-connectivity" } 09/01/2025 01:51:49.026 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-storage-cluster-state config_type = alarm operation = UPDATE config = { uuid : "f0a932a7-ed9e-4b6d-8205-af5a9bfaa9cd"fq_name : ["default-global-system-config", "system-defined-storage-cluster-state"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["storage-cluster"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "StorageCluster.info_stats.status", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17341447536994306925, "uuid_lslong": 9369087403512998349}, "enable": true, "created": "2025-09-01T01:49:48.696473", "last_modified": "2025-09-01T01:49:48.696473", "description": "Storage Cluster warning/errors.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-storage-cluster-state" } 09/01/2025 01:51:49.027 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-interface config_type = alarm operation = UPDATE config = { uuid : "06f05838-c6e1-4e04-ac18-6ecbd1e550d6"fq_name : ["default-global-system-config", "system-defined-vrouter-interface"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": ">=", "operand1": "VrouterAgent.down_interface_count", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 499996559516192260, "uuid_lslong": 12400783395643543766}, "enable": true, "created": "2025-09-01T01:49:48.759522", "last_modified": "2025-09-01T01:49:48.759522", "description": "Vrouter interface(s) down.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-interface" } 09/01/2025 01:51:49.027 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down config_type = alarm operation = UPDATE config = { uuid : "5c2f1f41-e227-45e7-bcb9-4f16572822a7"fq_name : ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.l3mh_phy_interface_down", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6642562343222330855, "uuid_lslong": 13598987507192046247}, "enable": true, "created": "2025-09-01T01:49:49.260104", "last_modified": "2025-09-01T01:49:49.260104", "description": "On l3 multihomed compute one or more physical interfaces are Inactive.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-l3mh-physical-interface-link-down" } 09/01/2025 01:51:49.027 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-limit-exceeded config_type = alarm operation = UPDATE config = { uuid : "1e6bd688-05b0-46c8-84fd-ee8c98e33e0b"fq_name : ["default-global-system-config", "system-defined-vrouter-limit-exceeded"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.res_limit", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 2192081523345213128, "uuid_lslong": 9583077869742145035}, "enable": true, "created": "2025-09-01T01:49:49.135034", "last_modified": "2025-09-01T01:49:49.135034", "description": "Agent resource usage exceeded configured watermark for resource.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-limit-exceeded" } 09/01/2025 01:51:49.027 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-table-limit-exceeded config_type = alarm operation = UPDATE config = { uuid : "05605c78-eb1e-46cf-8c4f-9b21ef8e2040"fq_name : ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.res_table_limit", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 387411242364323535, "uuid_lslong": 10110470258525347904}, "enable": true, "created": "2025-09-01T01:49:49.194896", "last_modified": "2025-09-01T01:49:49.194896", "description": "Agent resource usage exceeded table size for resource in vrouter.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-table-limit-exceeded" } 09/01/2025 01:51:49.027 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-xmpp-close-reason config_type = alarm operation = UPDATE config = { uuid : "f90fdb0a-4813-4f62-ab83-06544c911831"fq_name : ["default-global-system-config", "system-defined-xmpp-close-reason"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["xmpp-peer"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "XmppPeerInfoData.close_reason", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "!=", "operand1": "XmppPeerInfoData.state_info.state", "operand2": {"uve_attribute": null, "json_value": "\"Established\""}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17946803877300096866, "uuid_lslong": 12358728761566369841}, "enable": true, "created": "2025-09-01T01:49:48.884641", "last_modified": "2025-09-01T01:49:48.884641", "description": "XMPP connection closed towards peer, alarm has close reason", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-xmpp-close-reason" } 09/01/2025 01:51:49.028 7fc2ad7d8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-xmpp-connectivity config_type = alarm operation = UPDATE config = { uuid : "9c892047-c967-4785-b950-784a14f751c3"fq_name : ["default-global-system-config", "system-defined-xmpp-connectivity"]parent_type : "global-system-config"parent_uuid : "ee2862bf-bd6e-4089-89a5-6a39d8597c3b"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "BgpRouterState.num_up_xmpp_peer", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "BgpRouterState.num_up_xmpp_peer", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "reader", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11279582234419873669, "uuid_lslong": 13353305154728186307}, "enable": true, "created": "2025-09-01T01:49:48.820365", "last_modified": "2025-09-01T01:49:48.820365", "description": "XMPP peer mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-xmpp-connectivity" } 09/01/2025 01:51:49.878 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:51:50.880 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:51:51.881 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:51:52.875 7fc2ad615680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:52.876 7fc2ad615680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:52.877 7fc2ad615680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:52.877 7fc2ad615680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:52.877 7fc2ad615680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:52.877 7fc2ad615680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:52.877 7fc2ad615680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:52.878 7fc2ad615680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:52.878 7fc2ad615680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:52.878 7fc2ad615680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:52.878 7fc2ad615680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:52.878 7fc2ad615680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:52.878 7fc2ad615680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:52.878 7fc2ad615680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:52.882 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:51:53.558 7fc2ad62f900 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3994-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.33', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3994-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3994-1.'], 'id_perms': {'created': '2025-09-01T01:51:53.551485', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:51:53.551485', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10223489193050112955, 'uuid_mslong': 14931147901897818605}}, 'parent_type': 'global-system-config', 'parent_uuid': 'ee2862bf-bd6e-4089-89a5-6a39d8597c3b', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'cf3617e8-7d70-41ed-8de1-214134cc03bb'}, 'oper': 'CREATE', 'request-id': 'req-8a8a7909-ce0e-40ea-ad1a-4f3dbd336145', 'type': 'analytics_snmp_node', 'uuid': 'cf3617e8-7d70-41ed-8de1-214134cc03bb'} 09/01/2025 01:51:53.559 7fc2ad62f900 [contrail-alarm-gen] [ERROR]: Error in rabbitmq drainer greenlet: 'AnalyticsLogger' object has no attribute 'warn' 09/01/2025 01:51:53.559 7fc2ad62f900 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:51:53.559 7fc2ad615680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:53.559 7fc2ad615680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:51:53.559 7fc2ad615680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:53.562 7fc2ad62f900 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:51:53.562 7fc2ad615680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:53.563 7fc2ad615680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.33:6379 server_addrs = [ 10.0.0.33:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:53.563 7fc2ad615680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:53.884 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:51:53.947 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:53.947 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:53.947 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:54.885 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:51:55.887 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:51:56.878 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:56.879 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:56.879 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:56.879 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:56.879 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:56.879 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:56.879 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:56.879 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:56.879 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:56.880 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:56.880 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:56.880 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:56.880 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:56.880 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:56.888 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:51:57.889 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:51:58.893 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:51:59.894 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:00.880 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:00.881 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:00.881 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:00.881 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:00.881 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:00.881 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:00.881 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:00.882 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:00.882 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:00.882 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:00.882 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:00.882 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:00.882 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:00.882 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:00.895 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:01.897 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:02.899 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:03.900 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:04.882 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:04.883 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:04.884 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:04.884 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:04.884 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:04.884 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:04.884 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:04.884 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:04.884 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:04.884 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:04.884 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:04.885 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:04.885 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:04.885 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:04.901 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:05.902 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:06.903 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:07.904 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:08.884 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:08.885 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:08.885 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:08.885 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:08.885 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:08.885 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:08.885 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:08.886 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:08.886 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:08.886 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:08.886 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:08.886 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:08.886 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:08.886 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:08.904 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:09.906 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:10.908 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:11.909 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:12.886 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:12.887 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:12.887 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:12.887 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:12.887 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:12.888 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:12.888 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:12.888 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:12.888 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:12.888 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:12.888 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:12.888 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:12.888 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:12.888 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:12.911 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:13.912 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:14.913 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:15.914 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:16.888 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:16.889 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:16.889 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:16.889 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:16.889 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:16.889 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:16.889 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:16.890 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:16.890 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:16.890 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:16.890 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:16.890 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:16.890 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:16.890 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:16.915 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:17.916 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:18.918 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:19.920 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:20.890 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:20.891 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:20.891 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:20.891 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:20.891 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:20.891 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:20.891 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:20.892 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:20.892 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:20.892 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:20.892 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:20.892 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:20.892 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:20.892 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:20.921 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:21.923 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:22.924 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:23.928 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:24.892 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:24.894 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:24.894 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:24.894 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:24.894 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:24.895 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:24.895 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:24.895 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:24.895 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:24.895 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:24.895 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:24.896 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:24.896 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:24.896 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:24.929 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:25.931 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:26.932 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:27.934 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:28.895 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:28.896 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:28.896 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:28.896 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:28.896 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:28.896 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:28.896 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:28.897 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:28.897 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:28.897 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:28.897 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:28.897 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:28.897 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:28.897 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:28.935 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:29.936 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:30.938 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:31.940 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:32.897 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:32.898 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:32.898 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:32.898 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:32.898 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:32.898 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:32.898 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:32.898 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:32.898 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:32.898 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:32.899 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:32.899 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:32.899 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:32.899 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:32.941 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:33.942 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:34.944 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:35.946 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:36.899 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:36.900 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:36.901 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:36.901 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:36.901 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:36.901 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:36.901 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:36.901 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:36.901 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:36.902 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:36.902 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:36.902 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:36.902 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:36.902 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:36.947 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:37.948 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:38.950 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:39.951 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:40.902 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:40.903 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:40.903 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:40.903 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:40.903 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:40.903 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:40.903 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:40.904 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:40.904 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:40.904 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:40.904 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:40.904 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:40.904 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:40.904 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:40.953 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:41.954 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:42.955 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:43.956 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:44.903 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:44.905 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:44.905 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:44.905 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:44.905 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:44.905 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:44.905 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:44.905 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:44.906 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:44.906 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:44.906 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:44.906 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:44.906 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:44.906 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:44.957 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:45.959 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:46.960 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:47.962 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:48.877 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:52:48.877 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:48.878 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:52:48.878 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:48.905 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:48.906 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:48.906 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:48.906 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:48.906 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:48.907 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:48.907 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:48.907 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:48.907 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:48.907 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:48.907 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:48.907 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:48.907 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:48.907 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:48.963 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:49.965 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:50.966 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:51.968 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:52.907 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:52.909 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:52.909 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:52.909 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:52.909 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:52.909 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:52.909 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:52.910 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:52.910 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:52.910 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:52.910 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:52.910 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:52.910 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:52.910 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:52.969 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:53.970 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:54.971 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:55.972 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:56.910 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:56.911 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:56.911 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:56.911 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:56.911 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:56.911 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:56.911 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:56.912 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:56.912 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:56.912 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:56.912 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:56.912 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:56.912 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:56.912 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:56.972 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:57.973 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:58.974 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:52:59.975 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:00.912 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:00.913 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:00.913 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:00.913 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:00.913 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:00.913 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:00.913 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:00.914 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:00.914 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:00.914 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:00.914 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:00.914 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:00.914 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:00.914 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:00.976 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:01.978 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:02.979 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:03.980 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:04.914 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:04.915 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:04.915 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:04.915 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:04.915 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:04.915 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:04.915 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:04.916 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:04.916 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:04.916 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:04.916 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:04.916 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:04.916 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:04.916 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:04.981 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:05.983 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:06.984 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:07.985 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:08.916 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:08.917 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:08.918 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:08.918 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:08.918 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:08.918 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:08.918 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:08.918 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:08.918 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:08.918 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:08.918 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:08.918 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:08.919 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:08.919 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:08.986 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:09.988 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:10.989 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:11.991 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:12.919 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:12.920 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:12.920 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:12.920 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:12.920 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:12.920 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:12.920 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:12.921 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:12.921 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:12.921 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:12.921 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:12.922 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:12.922 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:12.922 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:12.992 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:13.994 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:14.995 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:15.996 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:16.922 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:16.923 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:16.923 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:16.923 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:16.923 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:16.923 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:16.923 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:16.923 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:16.924 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:16.924 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:16.924 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:16.924 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:16.924 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:16.924 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:16.997 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:17.999 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:19.000 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:20.001 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:20.924 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:20.925 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:20.925 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:20.925 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:20.925 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:20.926 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:20.926 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:20.926 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:20.926 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:20.926 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:20.926 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:20.926 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:20.926 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:20.926 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:21.002 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:22.003 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:23.004 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:24.005 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:24.927 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:24.928 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:24.928 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:24.928 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:24.928 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:24.928 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:24.928 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:24.928 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:24.928 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:24.929 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:24.929 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:24.929 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:24.929 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:24.929 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:25.006 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:26.008 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:27.010 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:28.011 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:28.929 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:28.930 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:28.930 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:28.930 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:28.930 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:28.930 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:28.930 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:28.931 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:28.931 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:28.931 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:28.931 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:28.931 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:28.931 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:28.931 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:29.013 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:30.015 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:31.016 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:32.017 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:32.931 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:32.932 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:32.932 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:32.932 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:32.933 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:32.933 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:32.933 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:32.933 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:32.934 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:32.934 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:32.934 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:32.934 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:32.934 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:32.934 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:33.017 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:34.018 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:35.020 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:36.022 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:36.934 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:36.935 7fc2ad615860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:36.935 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:36.935 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:36.935 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:36.935 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:36.935 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:36.936 7fc2ad615860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:36.936 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:36.936 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:36.936 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:36.936 7fc2ad615860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:36.936 7fc2ad615860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:36.936 7fc2ad615860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:37.023 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:38.025 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:39.026 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:40.028 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:40.936 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:40.937 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:40.937 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:40.937 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:40.937 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:40.937 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:40.937 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:40.938 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:40.938 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:40.938 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:40.938 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:40.938 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:40.938 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:40.938 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:41.029 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:42.031 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:43.033 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:44.034 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:44.937 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:44.938 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:44.939 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:44.939 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:44.939 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:44.939 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:44.939 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:44.939 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:44.939 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:44.940 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:44.940 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:44.940 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:44.940 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:44.940 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:45.035 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:46.037 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:47.038 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:48.040 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:48.878 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:53:48.879 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:48.879 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:53:48.879 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:48.939 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:48.940 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:48.940 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:48.941 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:48.941 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:48.941 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:48.941 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:48.941 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:48.941 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:48.941 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:48.941 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:48.941 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:48.942 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:48.942 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:49.040 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:50.042 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:51.043 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:52.045 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:52.941 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:52.942 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:52.943 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:52.943 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:52.943 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:52.943 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:52.943 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:52.943 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:52.943 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:52.943 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:52.944 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:52.944 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:52.944 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:52.944 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:53.046 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:54.047 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:55.048 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:56.050 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:56.944 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:56.945 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:56.945 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:56.945 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:56.946 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:56.946 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:56.946 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:56.946 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:56.946 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:56.946 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:56.946 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:56.946 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:56.946 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:56.946 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:57.051 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:58.052 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:53:59.053 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:00.055 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:00.946 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:00.947 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:00.947 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:00.947 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:00.948 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:00.948 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:00.948 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:00.948 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:00.948 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:00.948 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:00.948 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:00.948 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:00.948 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:00.948 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:01.057 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:02.058 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:03.060 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:04.061 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:04.948 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:04.949 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:04.949 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:04.949 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:04.949 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:04.950 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:04.950 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:04.950 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:04.950 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:04.950 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:04.950 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:04.950 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:04.950 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:04.950 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:05.062 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:06.064 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:07.065 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:08.066 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:08.950 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:08.951 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:08.951 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:08.951 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:08.951 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:08.951 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:08.951 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:08.952 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:08.952 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:08.952 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:08.952 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:08.952 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:08.952 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:08.952 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:09.068 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:10.069 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:11.070 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:12.072 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:12.952 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:12.959 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:12.959 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:12.960 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:12.960 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:12.960 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:12.960 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:12.961 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:12.961 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:12.961 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:12.961 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:12.961 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:12.962 7fc2ad644a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:12.962 7fc2ad644a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:13.073 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:14.074 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:15.076 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:16.077 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:16.383 7fc2ad62f900 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-f40374ec-a407-4671-ab44-7c7d794a0d7a', 'type': 'global_system_config', 'uuid': 'ee2862bf-bd6e-4089-89a5-6a39d8597c3b'} 09/01/2025 01:54:16.500 7fc2ad62f900 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3994-1'], 'obj_dict': {'display_name': 'cn-jenkins-deploy-platform-ansible-os-3994-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3994-1'], 'id_perms': {'created': '2025-09-01T01:54:16.493303', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:54:16.493303', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10603178825188901869, 'uuid_mslong': 3986343829420525011}}, 'parent_type': 'global-system-config', 'parent_uuid': 'ee2862bf-bd6e-4089-89a5-6a39d8597c3b', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '37525697-a12e-45d3-9326-0ef6a55403ed', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.254'}, 'oper': 'CREATE', 'request-id': 'req-71effe9d-df46-4c58-885e-639891b4fe31', 'type': 'virtual_router', 'uuid': '37525697-a12e-45d3-9326-0ef6a55403ed'} 09/01/2025 01:54:16.804 7fc2ad62f900 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3994-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3994-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T01:54:16.797916', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:54:16.797916', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10931294390135619323, 'uuid_mslong': 2932886037452703012}}, 'parent_type': 'virtual-router', 'parent_uuid': '37525697-a12e-45d3-9326-0ef6a55403ed', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '28b3b44f-dd12-4524-97b3-c254d340fefb', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:28:b3:b4:4f:dd']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'd448dc3a-db1a-4974-b6b8-d1b35397cd6c'}]}, 'oper': 'CREATE', 'request-id': 'req-32170e8b-5561-45f2-a992-d175e3c85966', 'type': 'virtual_machine_interface', 'uuid': '28b3b44f-dd12-4524-97b3-c254d340fefb'} 09/01/2025 01:54:16.815 7fc2ad62f900 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3994-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-32170e8b-5561-45f2-a992-d175e3c85966', 'type': 'virtual_machine_interface', 'uuid': '28b3b44f-dd12-4524-97b3-c254d340fefb'} 09/01/2025 01:54:16.959 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:16.960 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:16.961 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:16.961 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:16.961 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:16.961 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:16.961 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:16.961 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:16.961 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:16.962 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:16.962 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:16.962 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:16.962 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:16.962 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:17.078 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:17.326 7fc2ad62f900 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-c6049cce-a974-4bd9-9bd4-986b4ea6a7ae', 'type': 'access_control_list', 'uuid': '9ec7ca98-e500-4c4f-93f9-50bddc16e41f'} 09/01/2025 01:54:18.080 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:19.081 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:20.082 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:20.962 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:20.963 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:20.963 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:20.964 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:20.964 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:20.964 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:20.964 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:20.964 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:20.964 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:20.964 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:20.964 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:20.965 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:20.965 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:20.965 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:21.083 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:21.476 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:54:22.085 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:23.087 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:24.087 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:24.965 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:24.966 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:24.967 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:24.967 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:24.967 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:24.967 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:24.967 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:24.967 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:24.967 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:24.968 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:24.968 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:24.968 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:24.968 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:24.968 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:25.089 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:26.091 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:27.092 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:28.093 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:28.968 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:28.969 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:28.969 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:28.969 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:28.969 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:28.969 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:28.969 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:28.970 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:28.970 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:28.970 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:28.970 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:28.970 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:28.970 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:28.970 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:29.094 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:30.096 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:31.097 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:32.099 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:32.970 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:32.971 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:32.971 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:32.971 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:32.971 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:32.971 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:32.971 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:32.972 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:32.972 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:32.972 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:32.972 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:32.972 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:32.972 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:32.972 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:33.100 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:33.528 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:54:34.101 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:35.103 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:36.105 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:36.972 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:36.973 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:36.973 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:36.973 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:36.973 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:36.974 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:36.974 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:36.974 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:36.974 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:36.974 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:36.974 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:36.974 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:36.974 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:36.974 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:37.106 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:38.106 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:39.108 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:40.109 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:40.975 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:40.976 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:40.976 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:40.976 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:40.976 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:40.976 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:40.976 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:40.976 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:40.977 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:40.977 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:40.977 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:40.977 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:40.977 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:40.977 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:41.110 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:42.112 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:43.114 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:44.115 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:44.976 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:44.977 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:44.977 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:44.978 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:44.978 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:44.978 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:44.978 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:44.978 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:44.978 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:44.978 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:44.978 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:44.978 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:44.978 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:44.979 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:45.116 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:45.519 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:54:46.117 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:47.119 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:48.121 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:48.880 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:54:48.880 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:48.881 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:54:48.881 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:48.978 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:48.979 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:48.979 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:48.979 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:48.980 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:48.980 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:48.980 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:48.980 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:48.980 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:48.980 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:48.980 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:48.980 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:48.980 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:48.980 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:49.122 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:50.124 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:51.126 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:52.127 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:52.980 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:52.981 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:52.982 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:52.982 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:52.982 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:52.982 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:52.982 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:52.982 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:52.982 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:52.982 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:52.982 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:52.982 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:52.983 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:52.983 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:53.128 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:54.129 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:55.130 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:56.132 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:56.983 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:56.984 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:56.984 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:56.984 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:56.984 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:56.984 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:56.984 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:56.985 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:56.985 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:56.985 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:56.985 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:56.985 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:56.985 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:56.985 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:57.132 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:57.578 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:54:58.134 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:54:59.135 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:00.137 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:00.985 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:00.986 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:00.986 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:00.986 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:00.987 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:00.987 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:00.987 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:00.987 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:00.987 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:00.987 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:00.987 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:00.987 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:00.987 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:00.987 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:01.138 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:02.139 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:03.141 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:04.142 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:04.988 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:04.989 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:04.989 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:04.989 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:04.989 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:04.989 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:04.989 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:04.989 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:04.990 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:04.990 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:04.990 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:04.990 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:04.990 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:04.990 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:05.143 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:06.144 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:07.146 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:08.148 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:08.990 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:08.991 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:08.991 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:08.991 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:08.991 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:08.991 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:08.991 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:08.991 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:08.991 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:08.992 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:08.992 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:08.992 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:08.992 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:08.992 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:09.149 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:09.652 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:55:10.151 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:11.153 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:12.155 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:12.993 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:12.994 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:12.994 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:12.994 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:12.994 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:12.994 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:12.994 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:12.994 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:12.995 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:12.995 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:12.995 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:12.995 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:12.995 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:12.995 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:13.156 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:14.158 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:15.159 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:16.160 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:16.995 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:16.996 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:16.996 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:16.996 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:16.996 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:16.996 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:16.996 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:16.997 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:16.997 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:16.997 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:16.997 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:16.997 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:16.997 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:16.997 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:17.162 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:18.164 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:19.165 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:20.166 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:20.997 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:20.998 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:20.998 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:20.998 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:20.999 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:20.999 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:20.999 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:20.999 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:20.999 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:20.999 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:20.999 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:20.999 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:20.999 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:20.999 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:21.168 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:21.975 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:55:22.169 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:23.170 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:24.171 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:25.000 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:25.001 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:25.001 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:25.001 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:25.001 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:25.001 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:25.001 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:25.002 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:25.002 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:25.002 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:25.002 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:25.002 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:25.002 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:25.002 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:25.173 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:26.174 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:27.176 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:28.177 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:29.001 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:29.002 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:29.003 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:29.003 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:29.003 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:29.003 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:29.003 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:29.003 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:29.003 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:29.003 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:29.004 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:29.004 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:29.004 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:29.004 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:29.178 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:30.180 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:31.181 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:32.182 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:33.004 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:33.005 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:33.005 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:33.005 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:33.005 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:33.005 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:33.005 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:33.006 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:33.006 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:33.006 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:33.006 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:33.006 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:33.006 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:33.006 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:33.183 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:34.007 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:55:34.185 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:35.187 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:36.188 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:37.006 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:37.007 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:37.007 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:37.007 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:37.007 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:37.008 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:37.008 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:37.008 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:37.008 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:37.008 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:37.008 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:37.008 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:37.008 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:37.008 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:37.190 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:38.191 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:39.192 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:40.194 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:41.008 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:41.009 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:41.010 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:41.010 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:41.010 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:41.010 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:41.010 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:41.010 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:41.010 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:41.010 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:41.010 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:41.010 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:41.011 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:41.011 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:41.195 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:42.197 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:43.198 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:44.200 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:45.010 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:45.011 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:45.011 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:45.011 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:45.012 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:45.012 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:45.012 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:45.012 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:45.012 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:45.012 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:45.012 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:45.012 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:45.012 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:45.013 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:45.201 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:45.987 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:55:46.202 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:47.204 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:48.206 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:48.881 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:55:48.881 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:48.882 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:55:48.882 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:49.012 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:49.013 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:49.013 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:49.013 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:49.013 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:49.013 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:49.013 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:49.013 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:49.014 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:49.014 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:49.014 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:49.014 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:49.014 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:49.014 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:49.207 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:50.209 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:51.211 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:52.212 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:53.014 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:53.015 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:53.016 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:53.016 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:53.016 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:53.016 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:53.016 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:53.017 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:53.017 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:53.017 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:53.017 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:53.017 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:53.017 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:53.017 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:53.213 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:54.214 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:55.215 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:56.217 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:57.017 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:57.018 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:57.018 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:57.018 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:57.018 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:57.018 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:57.018 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:57.019 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:57.019 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:57.019 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:57.019 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:57.019 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:57.019 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:57.019 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:57.218 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:58.053 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:55:58.219 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:55:59.223 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:00.225 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:01.020 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:01.021 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:01.021 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:01.021 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:01.022 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:01.022 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:01.022 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:01.022 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:01.022 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:01.022 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:01.022 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:01.022 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:01.023 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:01.023 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:01.227 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:02.229 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:03.230 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:04.232 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:05.023 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:05.024 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:05.024 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:05.024 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:05.024 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:05.024 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:05.025 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:05.025 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:05.025 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:05.025 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:05.025 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:05.025 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:05.025 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:05.025 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:05.234 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:06.236 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:07.237 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:08.239 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:09.025 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:09.026 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:09.026 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:09.026 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:09.026 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:09.026 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:09.026 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:09.027 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:09.027 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.027 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:09.027 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.027 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.027 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:09.027 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:09.240 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:10.131 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:56:10.241 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:11.243 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:12.245 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:13.027 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:13.028 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:13.029 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:13.029 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:13.029 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:13.029 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:13.029 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:13.029 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:13.029 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.029 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:13.029 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.029 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.030 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:13.030 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:13.246 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:14.248 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:15.250 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:16.251 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:17.030 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:17.031 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:17.031 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:17.031 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:17.031 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:17.031 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:17.032 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:17.032 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:17.032 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:17.032 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:17.032 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:17.032 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:17.032 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:17.032 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:17.253 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:18.255 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:19.256 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:20.257 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:21.032 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:21.033 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:21.033 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:21.033 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:21.033 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:21.033 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:21.033 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:21.034 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:21.034 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:21.034 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:21.034 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:21.034 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:21.034 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:21.034 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:21.258 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:22.260 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:22.483 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:56:23.261 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:24.262 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:25.034 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:25.035 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:25.035 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:25.035 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:25.035 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:25.035 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:25.036 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:25.036 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:25.036 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:25.036 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:25.036 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:25.036 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:25.036 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:25.036 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:25.264 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:26.265 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:27.267 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:28.269 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:29.036 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:29.037 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:29.037 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:29.037 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:29.038 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:29.038 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:29.038 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:29.038 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:29.038 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:29.038 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:29.038 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:29.038 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:29.038 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:29.038 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:29.270 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:30.272 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:31.273 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:32.274 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:33.038 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:33.040 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:33.040 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:33.040 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:33.040 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:33.040 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:33.040 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:33.041 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:33.041 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:33.041 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:33.041 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:33.041 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:33.041 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:33.041 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:33.276 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:34.277 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:34.504 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:56:35.279 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:36.280 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:37.040 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:37.042 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:37.042 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:37.042 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:37.042 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:37.042 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:37.042 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:37.043 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:37.043 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:37.043 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:37.043 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:37.043 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:37.043 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:37.043 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:37.281 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:38.283 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:39.284 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:40.286 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:41.043 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:41.044 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:41.045 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:41.045 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:41.045 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:41.045 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:41.045 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:41.045 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:41.045 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:41.046 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:41.046 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:41.046 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:41.046 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:41.046 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:41.287 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:42.289 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:43.291 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:44.293 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:45.046 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:45.047 7fc2ad644680 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:45.047 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:45.047 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:45.048 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:45.048 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:45.048 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:45.048 7fc2ad644680 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:45.048 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:45.048 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:45.048 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:45.048 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:45.048 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:45.049 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:45.294 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:46.295 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:46.487 7fc2ad644680 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:56:47.297 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:48.299 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:48.878 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Starting the Kafka producer 09/01/2025 01:56:48.879 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name connections-closed 09/01/2025 01:56:48.879 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name connections-created 09/01/2025 01:56:48.879 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name select-time 09/01/2025 01:56:48.879 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name io-time 09/01/2025 01:56:48.879 7fc2ad71ba40 [kafka.client] [DEBUG]: Attempting to check version with node bootstrap-0 09/01/2025 01:56:48.879 7fc2ad71ba40 [kafka.client] [DEBUG]: Initiating connection to node bootstrap-0 at 10.0.0.33:9092 09/01/2025 01:56:48.880 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-sent-received 09/01/2025 01:56:48.880 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-sent 09/01/2025 01:56:48.880 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-received 09/01/2025 01:56:48.880 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name request-latency 09/01/2025 01:56:48.880 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name throttle-time 09/01/2025 01:56:48.880 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.bytes-sent 09/01/2025 01:56:48.880 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.bytes-received 09/01/2025 01:56:48.880 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.latency 09/01/2025 01:56:48.880 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.throttle 09/01/2025 01:56:48.881 7fc2ad71ba40 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 01:56:48.881 7fc2ad71ba40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 01:56:48.881 7fc2ad71ba40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.33', 9092)]>: connecting to 10.0.0.33:9092 [('10.0.0.33', 9092) IPv4] 09/01/2025 01:56:48.881 7fc2ad71ba40 [kafka.client] [DEBUG]: Timeouts: user 199.970245, metadata inf, idle connection inf, request inf 09/01/2025 01:56:48.882 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:56:48.882 7fc2ad71ba40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: established TCP connection 09/01/2025 01:56:48.882 7fc2ad71ba40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: checking broker Api Versions 09/01/2025 01:56:48.882 7fc2ad71ba40 [kafka.protocol.parser] [DEBUG]: Sending request ApiVersionsRequest_v4(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 01:56:48.882 7fc2ad71ba40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 1 (timeout_ms 1600.0): ApiVersionsRequest_v4(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 01:56:48.883 7fc2ad71ba40 [kafka.client] [DEBUG]: Timeouts: user 199.978352, metadata inf, idle connection inf, request 1599.733114 09/01/2025 01:56:48.883 7fc2ad644680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:48.883 7fc2ad644680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:56:48.883 7fc2ad644680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:48.888 7fc2ad71ba40 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 01:56:48.888 7fc2ad71ba40 [kafka.protocol.parser] [DEBUG]: Processing response ApiVersionsResponse_v4 09/01/2025 01:56:48.888 7fc2ad71ba40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 1 (6.101131439208984 ms): ApiVersionsResponse_v0(error_code=35, api_versions=[(api_key=18, min_version=0, max_version=3)]) 09/01/2025 01:56:48.889 7fc2ad71ba40 [kafka.client] [DEBUG]: Timeouts: user 199.979782, metadata inf, idle connection 539990.000000, request inf 09/01/2025 01:56:48.889 7fc2ad71ba40 [kafka.protocol.parser] [DEBUG]: Sending request ApiVersionsRequest_v3(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 01:56:48.889 7fc2ad71ba40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 2 (timeout_ms 1600.0): ApiVersionsRequest_v3(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 01:56:48.889 7fc2ad71ba40 [kafka.client] [DEBUG]: Timeouts: user 199.980259, metadata inf, idle connection 539989.000000, request 1599.787474 09/01/2025 01:56:48.891 7fc2ad71ba40 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 01:56:48.891 7fc2ad71ba40 [kafka.protocol.parser] [DEBUG]: Processing response ApiVersionsResponse_v3 09/01/2025 01:56:48.891 7fc2ad71ba40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 2 (2.3796558380126953 ms): ApiVersionsResponse_v3(error_code=0, api_versions=[(api_key=0, min_version=0, max_version=8, _tagged_fields={}), (api_key=1, min_version=0, max_version=11, _tagged_fields={}), (api_key=2, min_version=0, max_version=5, _tagged_fields={}), (api_key=3, min_version=0, max_version=9, _tagged_fields={}), (api_key=4, min_version=0, max_version=4, _tagged_fields={}), (api_key=5, min_version=0, max_version=3, _tagged_fields={}), (api_key=6, min_version=0, max_version=6, _tagged_fields={}), (api_key=7, min_version=0, max_version=3, _tagged_fields={}), (api_key=8, min_version=0, max_version=8, _tagged_fields={}), (api_key=9, min_version=0, max_version=7, _tagged_fields={}), (api_key=10, min_version=0, max_version=3, _tagged_fields={}), (api_key=11, min_version=0, max_version=7, _tagged_fields={}), (api_key=12, min_version=0, max_version=4, _tagged_fields={}), (api_key=13, min_version=0, max_version=4, _tagged_fields={}), (api_key=14, min_version=0, max_version=5, _tagged_fields={}), (api_key=15, min_version=0, max_version=5, _tagged_fields={}), (api_key=16, min_version=0, max_version=4, _tagged_fields={}), (api_key=17, min_version=0, max_version=1, _tagged_fields={}), (api_key=18, min_version=0, max_version=3, _tagged_fields={}), (api_key=19, min_version=0, max_version=5, _tagged_fields={}), (api_key=20, min_version=0, max_version=4, _tagged_fields={}), (api_key=21, min_version=0, max_version=2, _tagged_fields={}), (api_key=22, min_version=0, max_version=3, _tagged_fields={}), (api_key=23, min_version=0, max_version=3, _tagged_fields={}), (api_key=24, min_version=0, max_version=1, _tagged_fields={}), (api_key=25, min_version=0, max_version=1, _tagged_fields={}), (api_key=26, min_version=0, max_version=1, _tagged_fields={}), (api_key=27, min_version=0, max_version=0, _tagged_fields={}), (api_key=28, min_version=0, max_version=3, _tagged_fields={}), (api_key=29, min_version=0, max_version=2, _tagged_fields={}), (api_key=30, min_version=0, max_version=2, _tagged_fields={}), (api_key=31, min_version=0, max_version=2, _tagged_fields={}), (api_key=32, min_version=0, max_version=3, _tagged_fields={}), (api_key=33, min_version=0, max_version=1, _tagged_fields={}), (api_key=34, min_version=0, max_version=1, _tagged_fields={}), (api_key=35, min_version=0, max_version=2, _tagged_fields={}), (api_key=36, min_version=0, max_version=2, _tagged_fields={}), (api_key=37, min_version=0, max_version=2, _tagged_fields={}), (api_key=38, min_version=0, max_version=2, _tagged_fields={}), (api_key=39, min_version=0, max_version=2, _tagged_fields={}), (api_key=40, min_version=0, max_version=2, _tagged_fields={}), (api_key=41, min_version=0, max_version=2, _tagged_fields={}), (api_key=42, min_version=0, max_version=2, _tagged_fields={}), (api_key=43, min_version=0, max_version=2, _tagged_fields={}), (api_key=44, min_version=0, max_version=1, _tagged_fields={}), (api_key=45, min_version=0, max_version=0, _tagged_fields={}), (api_key=46, min_version=0, max_version=0, _tagged_fields={}), (api_key=47, min_version=0, max_version=0, _tagged_fields={}), (api_key=48, min_version=0, max_version=0, _tagged_fields={}), (api_key=49, min_version=0, max_version=0, _tagged_fields={})], throttle_time_ms=0, _tagged_fields={}) 09/01/2025 01:56:48.892 7fc2ad71ba40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.33', 9092)]>: Broker version identified as 2.6 09/01/2025 01:56:48.892 7fc2ad71ba40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.33', 9092)]>: Connection complete. 09/01/2025 01:56:48.892 7fc2ad71ba40 [kafka.client] [DEBUG]: Node bootstrap-0 connected 09/01/2025 01:56:48.893 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name batch-size 09/01/2025 01:56:48.893 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name compression-rate 09/01/2025 01:56:48.893 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name queue-time 09/01/2025 01:56:48.893 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name records-per-request 09/01/2025 01:56:48.893 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes 09/01/2025 01:56:48.893 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name record-retries 09/01/2025 01:56:48.893 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name errors 09/01/2025 01:56:48.893 7fc2ad71ba40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name record-size-max 09/01/2025 01:56:48.893 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Starting Kafka producer I/O thread. 09/01/2025 01:56:48.894 7fc2ad644680 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) to node bootstrap-0 09/01/2025 01:56:48.894 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) 09/01/2025 01:56:48.894 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 3 (timeout_ms 30000): MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) 09/01/2025 01:56:48.894 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.655495, metadata 30000.000000, idle connection 539984.000000, request 29999.978781 09/01/2025 01:56:48.894 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Kafka producer started 09/01/2025 01:56:48.894 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: Initialized health-check KafkaProducer 09/01/2025 01:56:48.894 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: health-check KafkaProducer call returned 09/01/2025 01:56:48.894 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993811, metadata 30000.000000, idle connection 539984.000000, request 29999.521494 09/01/2025 01:56:48.901 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 01:56:48.901 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 01:56:48.901 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 3 (6.8645477294921875 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.33', port=9092, rack=None)], cluster_id='hYIJrOiPQM-832Z-Zin5Xw', controller_id=1, topics=[]) 09/01/2025 01:56:48.901 7fc2ad644680 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 0, groups: 0) 09/01/2025 01:56:48.901 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993095, metadata 299999.863281, idle connection 539977.000000, request inf 09/01/2025 01:56:49.048 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:49.049 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:49.049 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:49.049 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:49.049 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:49.049 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:49.049 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:49.050 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:49.050 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:49.050 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:49.050 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:49.050 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:49.050 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:49.050 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:49.300 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:50.301 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:51.302 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:52.304 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:53.050 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:53.051 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:53.051 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:53.051 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:53.052 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:53.052 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:53.052 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:53.052 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:53.052 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:53.052 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:53.052 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:53.052 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:53.053 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:53.053 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:53.305 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:54.306 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:55.308 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:56.309 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:57.053 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:57.054 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:57.054 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:57.054 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:57.055 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:57.055 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:57.055 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:57.055 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:57.055 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.055 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:57.055 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.055 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.056 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:57.056 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.310 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:58.312 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:56:58.545 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:56:59.313 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:00.315 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:01.056 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:01.057 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:01.057 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:01.057 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:01.057 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:01.057 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:01.057 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:01.058 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:01.058 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:01.058 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:01.058 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:01.058 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:01.058 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:01.058 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:01.316 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:02.318 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:03.320 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:04.322 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:05.058 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:05.059 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:05.059 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:05.059 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:05.059 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:05.060 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:05.060 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:05.060 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:05.060 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.060 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:05.060 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.060 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.060 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:05.060 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.323 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:06.325 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:07.326 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:08.328 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:09.060 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:09.061 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:09.061 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:09.061 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:09.062 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:09.062 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:09.062 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:09.062 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:09.062 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.062 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:09.062 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.062 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.062 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:09.062 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.329 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:10.331 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:10.587 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:57:11.332 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:12.334 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:13.063 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:13.064 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:13.064 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:13.064 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:13.064 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:13.064 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:13.064 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:13.065 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:13.065 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.065 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:13.065 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.065 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.065 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:13.065 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.336 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:14.337 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:15.338 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:16.340 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:17.065 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:17.066 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:17.066 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:17.066 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:17.067 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:17.067 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:17.067 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:17.067 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:17.067 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.067 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:17.067 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.067 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.067 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:17.067 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.341 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:18.343 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:19.344 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:20.346 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:21.067 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:21.069 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:21.069 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:21.069 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:21.069 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:21.069 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:21.069 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:21.069 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:21.069 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.070 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:21.070 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.070 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.070 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:21.070 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.347 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:22.349 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:22.937 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:57:23.351 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:24.352 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:25.070 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:25.071 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:25.072 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:25.072 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:25.072 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:25.072 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:25.072 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:25.072 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:25.072 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.073 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:25.073 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.073 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.073 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:25.073 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.353 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:26.355 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:27.357 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:28.358 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:29.072 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:29.074 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:29.074 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:29.074 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:29.074 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:29.074 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:29.075 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:29.075 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:29.075 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.075 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:29.075 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.075 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.076 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:29.076 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.360 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:30.362 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:31.364 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:32.365 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:33.075 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:33.076 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:33.076 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:33.076 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:33.076 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:33.076 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:33.076 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:33.076 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:33.077 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.077 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:33.077 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.077 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.077 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:33.077 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.366 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:34.368 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:35.030 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:57:35.369 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:36.371 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:37.077 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:37.078 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:37.078 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:37.078 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:37.078 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:37.078 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:37.078 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:37.078 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:37.079 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.079 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:37.079 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.079 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.079 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:37.079 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.372 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:38.374 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:39.375 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:40.377 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:41.078 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:41.079 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:41.080 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:41.080 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:41.080 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:41.080 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:41.080 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:41.080 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:41.080 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.080 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:41.080 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.081 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.081 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:41.081 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.378 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:42.380 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:43.381 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:44.382 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:45.081 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:45.082 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:45.083 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:45.083 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:45.083 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:45.083 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:45.083 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:45.083 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:45.083 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.083 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:45.084 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.084 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.084 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:45.084 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.383 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:46.385 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:47.087 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:57:47.386 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:48.388 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:48.883 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:57:48.883 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:48.884 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:57:48.884 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.084 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:49.085 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:49.085 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:49.085 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:49.085 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:49.085 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:49.085 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:49.085 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:49.085 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.086 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:49.086 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.086 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.086 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:49.086 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.389 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:50.391 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:51.393 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:52.394 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:53.086 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:53.087 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:53.087 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:53.087 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:53.087 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:53.087 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:53.088 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:53.088 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:53.088 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.088 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:53.088 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.088 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.088 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:53.088 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.395 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:54.396 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:55.397 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:56.398 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:57.088 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:57.089 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:57.089 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:57.089 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:57.090 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:57.090 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:57.090 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:57.090 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:57.090 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.090 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:57.090 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.090 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.090 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:57.090 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.400 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:58.402 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:57:59.035 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:57:59.403 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:00.405 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:01.090 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:01.092 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:01.092 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:01.092 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:01.092 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:01.092 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:01.092 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:01.092 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:01.093 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.093 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:01.093 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.093 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.093 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:01.093 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.406 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:02.408 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:03.410 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:04.412 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:05.093 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:05.094 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:05.094 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:05.094 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:05.095 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:05.095 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:05.095 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:05.095 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:05.095 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.095 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:05.095 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.095 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.095 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:05.095 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.413 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:06.415 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:07.417 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:08.418 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:09.095 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:09.096 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:09.096 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:09.096 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:09.097 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:09.097 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:09.097 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:09.097 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:09.097 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:09.097 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:09.097 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:09.097 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:09.097 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:09.097 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:09.420 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:10.420 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:11.129 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:58:11.422 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:12.424 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:13.097 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:13.098 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:13.098 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:13.098 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:13.099 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:13.099 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:13.099 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:13.099 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:13.099 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:13.099 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:13.099 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:13.099 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:13.099 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:13.100 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:13.425 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:14.427 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:15.429 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:16.430 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:17.099 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:17.100 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:17.101 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:17.101 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:17.101 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:17.101 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:17.101 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:17.101 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:17.101 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.101 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:17.102 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.102 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.102 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:17.102 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.432 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:18.433 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:19.434 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:20.435 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:21.102 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:21.103 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:21.103 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:21.103 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:21.103 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:21.103 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:21.103 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:21.103 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:21.104 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.104 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:21.104 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.104 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.104 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:21.104 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.436 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:22.438 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:23.439 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:23.490 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:58:24.441 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:25.104 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:25.105 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:25.106 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:25.106 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:25.106 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:25.106 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:25.106 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:25.106 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:25.106 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.106 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:25.106 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.107 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.107 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:25.107 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.442 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:26.443 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:27.445 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:28.447 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:29.106 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:29.107 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:29.108 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:29.108 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:29.108 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:29.108 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:29.108 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:29.108 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:29.108 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.109 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:29.109 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.109 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.109 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:29.109 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.448 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:30.449 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:31.451 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:32.452 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:33.108 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:33.110 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:33.110 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:33.110 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:33.110 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:33.110 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:33.110 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:33.110 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:33.111 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.111 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:33.111 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.111 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.111 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:33.111 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.453 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:34.455 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:35.456 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:35.499 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:58:36.458 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:37.111 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:37.112 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:37.112 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:37.112 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:37.113 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:37.113 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:37.113 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:37.113 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:37.113 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.113 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:37.113 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.113 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.113 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:37.113 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.459 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:38.460 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:39.462 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:40.464 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:41.114 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:41.115 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:41.115 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:41.115 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:41.115 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:41.115 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:41.115 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:41.116 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:41.116 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:41.116 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:41.116 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:41.116 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:41.116 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:41.116 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:41.465 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:42.467 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:43.468 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:44.469 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:45.116 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:45.137 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:45.137 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:45.138 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:45.138 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:45.138 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:45.138 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:45.138 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:45.138 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.138 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:45.138 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.138 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.139 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:45.139 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.470 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:46.472 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:47.473 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:47.508 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:58:48.474 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:48.883 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:58:48.884 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:48.884 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:58:48.884 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:48.895 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 01:58:48.896 7fc2ad644680 [kafka.client] [DEBUG]: Initializing connection to node 1 for metadata request 09/01/2025 01:58:48.896 7fc2ad644680 [kafka.client] [DEBUG]: Initiating connection to node 1 at 10.0.0.33:9092 09/01/2025 01:58:48.896 7fc2ad644680 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.bytes-sent 09/01/2025 01:58:48.896 7fc2ad644680 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.bytes-received 09/01/2025 01:58:48.897 7fc2ad644680 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.latency 09/01/2025 01:58:48.897 7fc2ad644680 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.throttle 09/01/2025 01:58:48.897 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 01:58:48.897 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 01:58:48.897 7fc2ad644680 [kafka.conn] [INFO]: [IPv4 ('10.0.0.33', 9092)]>: connecting to 10.0.0.33:9092 [('10.0.0.33', 9092) IPv4] 09/01/2025 01:58:48.897 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999988.365650, metadata inf, idle connection 419981.000000, request inf 09/01/2025 01:58:48.898 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: established TCP connection 09/01/2025 01:58:48.898 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: checking broker Api Versions 09/01/2025 01:58:48.898 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Using pre-configured api_version (2, 6) for ApiVersions 09/01/2025 01:58:48.898 7fc2ad644680 [kafka.conn] [INFO]: [IPv4 ('10.0.0.33', 9092)]>: Connection complete. 09/01/2025 01:58:48.898 7fc2ad644680 [kafka.client] [DEBUG]: Node 1 connected 09/01/2025 01:58:48.898 7fc2ad644680 [kafka.conn] [INFO]: [IPv4 ('10.0.0.33', 9092)]>: Closing connection. 09/01/2025 01:58:48.898 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: reconnect backoff 0.04780090005013805 after 1 failures 09/01/2025 01:58:48.898 7fc2ad644680 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 01:58:48.898 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 01:58:48.898 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 1 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 01:58:48.899 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.614248, metadata 30000.000000, idle connection 419980.000000, request 29999.977589 09/01/2025 01:58:48.899 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993572, metadata 30000.000000, idle connection 419979.000000, request 29999.685287 09/01/2025 01:58:48.936 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 01:58:48.936 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 01:58:48.937 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 1 (37.92285919189453 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.33', port=9092, rack=None)], cluster_id='hYIJrOiPQM-832Z-Zin5Xw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 01:58:48.937 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990950, metadata 99.963623, idle connection 419942.000000, request inf 09/01/2025 01:58:48.937 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.041724205017089844 secs. 09/01/2025 01:58:48.937 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 01:58:48.937 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993572, metadata 99.439453, idle connection 419941.000000, request inf 09/01/2025 01:58:49.038 7fc2ad644680 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 01:58:49.038 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 01:58:49.038 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 2 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 01:58:49.038 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.320278, metadata 30000.000000, idle connection 419840.000000, request 29999.959469 09/01/2025 01:58:49.039 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990711, metadata 30000.000000, idle connection 419840.000000, request 29999.461174 09/01/2025 01:58:49.043 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 01:58:49.044 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 01:58:49.044 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 2 (5.5084228515625 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.33', port=9092, rack=None)], cluster_id='hYIJrOiPQM-832Z-Zin5Xw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 01:58:49.044 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.989996, metadata 99.960449, idle connection 419834.000000, request inf 09/01/2025 01:58:49.044 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.14893460273742676 secs. 09/01/2025 01:58:49.044 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 01:58:49.045 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990473, metadata 99.357666, idle connection 419834.000000, request inf 09/01/2025 01:58:49.138 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:49.139 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:49.139 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:49.139 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:49.140 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:49.140 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:49.140 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:49.140 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:49.140 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.140 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:49.140 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.140 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.141 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:49.141 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.145 7fc2ad644680 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 01:58:49.145 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 01:58:49.145 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 3 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 01:58:49.145 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.526749, metadata 30000.000000, idle connection 419733.000000, request 29999.970913 09/01/2025 01:58:49.146 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.989042, metadata 30000.000000, idle connection 419732.000000, request 29999.518871 09/01/2025 01:58:49.150 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 01:58:49.150 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 01:58:49.150 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 3 (4.8885345458984375 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.33', port=9092, rack=None)], cluster_id='hYIJrOiPQM-832Z-Zin5Xw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 01:58:49.151 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993095, metadata 99.972900, idle connection 419728.000000, request inf 09/01/2025 01:58:49.151 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.25548386573791504 secs. 09/01/2025 01:58:49.151 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 01:58:49.151 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993334, metadata 99.522217, idle connection 419727.000000, request inf 09/01/2025 01:58:49.252 7fc2ad644680 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 01:58:49.252 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 01:58:49.252 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 4 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 01:58:49.252 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.396334, metadata 30000.000000, idle connection 419626.000000, request 29999.969959 09/01/2025 01:58:49.252 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992619, metadata 30000.000000, idle connection 419626.000000, request 29999.591351 09/01/2025 01:58:49.256 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 4 09/01/2025 01:58:49.256 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 01:58:49.256 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 4 (3.917694091796875 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.33', port=9092, rack=None)], cluster_id='hYIJrOiPQM-832Z-Zin5Xw', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 01:58:49.256 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992619, metadata 99.971680, idle connection 419622.000000, request inf 09/01/2025 01:58:49.256 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.3611750602722168 secs. 09/01/2025 01:58:49.256 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 01:58:49.257 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994287, metadata 99.554688, idle connection 419622.000000, request inf 09/01/2025 01:58:49.357 7fc2ad644680 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 01:58:49.358 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 01:58:49.358 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 5 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 01:58:49.358 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.408493, metadata 30000.000000, idle connection 419521.000000, request 29999.969721 09/01/2025 01:58:49.358 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993095, metadata 30000.000000, idle connection 419520.000000, request 29999.584913 09/01/2025 01:58:49.362 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 5 09/01/2025 01:58:49.362 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 01:58:49.362 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 5 (4.6443939208984375 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.33', port=9092, rack=None)], cluster_id='hYIJrOiPQM-832Z-Zin5Xw', controller_id=1, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=5, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=10, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=20, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=25, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=26, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=11, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=4, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=16, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=17, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=23, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=28, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=13, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=22, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=8, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=2, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=29, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=19, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=14, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=1, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=7, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[])])]) 09/01/2025 01:58:49.363 7fc2ad644680 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 1, groups: 0) 09/01/2025 01:58:49.363 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991903, metadata 299999.877930, idle connection 419515.000000, request inf 09/01/2025 01:58:49.364 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.46828770637512207 secs. 09/01/2025 01:58:49.364 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=18) 09/01/2025 01:58:49.364 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=18) is either full or getting a new batch 09/01/2025 01:58:49.364 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 01:58:49.364 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 01:58:49.364 7fc2ad71ba40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=18) 09/01/2025 01:58:49.364 7fc2ad644680 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.records-per-batch 09/01/2025 01:58:49.365 7fc2ad644680 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.bytes 09/01/2025 01:58:49.365 7fc2ad644680 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.compression-rate 09/01/2025 01:58:49.365 7fc2ad644680 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.record-retries 09/01/2025 01:58:49.365 7fc2ad644680 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.record-errors 09/01/2025 01:58:49.365 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 01:58:49.365 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xda\x9d\xa6\xa1\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x02\xff\x99\x14\x00\x00\x01\x99\x02\xff\x99\x14\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 01:58:49.365 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xda\x9d\xa6\xa1\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x02\xff\x99\x14\x00\x00\x01\x99\x02\xff\x99\x14\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 01:58:49.365 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xda\x9d\xa6\xa1\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x02\xff\x99\x14\x00\x00\x01\x99\x02\xff\x99\x14\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 01:58:49.365 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 6 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xda\x9d\xa6\xa1\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x02\xff\x99\x14\x00\x00\x01\x99\x02\xff\x99\x14\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 01:58:49.366 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 299997.683594, idle connection 419513.000000, request 29999.969721 09/01/2025 01:58:49.366 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993334, metadata 299997.332520, idle connection 419512.000000, request 29999.621630 09/01/2025 01:58:49.428 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 6 09/01/2025 01:58:49.428 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 01:58:49.428 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 6 (62.45899200439453 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 01:58:49.428 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=18, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 01:58:49.428 7fc2ad644680 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=18) with base offset 0 log start offset 0. 09/01/2025 01:58:49.429 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.989519, metadata 299934.655029, idle connection 419450.000000, request inf 09/01/2025 01:58:49.429 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 01:58:49.475 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:50.476 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:51.478 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:52.479 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:53.141 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:53.142 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:53.142 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:53.142 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:53.142 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:53.142 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:53.142 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:53.143 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:53.143 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.143 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:53.143 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.143 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.143 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:53.143 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.481 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:54.483 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:55.484 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:56.485 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:57.144 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:57.145 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:57.145 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:57.145 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:57.145 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:57.145 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:57.145 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:57.145 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:57.146 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.146 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:57.146 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.146 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.146 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:57.146 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.486 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:58.488 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:59.489 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:58:59.572 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:59:00.491 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:01.146 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:01.147 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:01.147 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:01.147 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:01.147 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:01.148 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:01.148 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:01.148 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:01.148 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.148 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:01.148 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.148 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.148 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:01.148 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.492 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:02.493 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:03.494 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:04.496 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:05.148 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:05.149 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:05.149 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:05.150 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:05.150 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:05.150 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:05.150 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:05.150 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:05.150 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.150 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:05.150 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.150 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.150 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:05.151 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.496 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:06.498 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:07.500 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:08.501 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:09.150 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:09.151 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:09.152 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:09.152 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:09.152 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:09.152 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:09.152 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:09.152 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:09.152 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.152 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:09.152 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.152 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.153 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:09.153 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.502 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:10.504 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:11.505 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:11.569 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:59:12.507 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:13.152 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:13.154 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:13.154 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:13.154 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:13.154 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:13.154 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:13.154 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:13.154 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:13.154 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:13.154 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:13.155 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:13.155 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:13.155 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:13.155 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:13.508 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:14.509 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:15.510 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:16.512 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:17.155 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:17.156 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:17.156 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:17.157 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:17.157 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:17.157 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:17.157 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:17.157 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:17.157 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.157 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:17.157 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.157 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.157 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:17.158 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.513 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:18.515 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:19.516 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:20.517 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:21.157 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:21.159 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:21.159 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:21.159 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:21.159 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:21.159 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:21.160 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:21.160 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:21.160 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.160 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:21.160 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.160 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.160 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:21.160 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.518 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:22.519 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:23.520 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:23.913 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:59:24.522 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:25.160 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:25.162 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:25.162 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:25.162 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:25.162 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:25.162 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:25.162 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:25.162 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:25.162 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.163 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:25.163 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.163 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.163 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:25.163 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.523 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:26.524 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:27.526 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:28.528 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:29.162 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:29.163 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:29.164 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:29.164 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:29.164 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:29.164 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:29.164 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:29.164 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:29.164 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.165 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:29.165 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.165 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.165 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:29.165 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.529 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:30.530 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:31.532 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:32.533 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:33.165 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:33.166 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:33.166 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:33.166 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:33.167 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:33.167 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:33.167 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:33.167 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:33.167 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.167 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:33.167 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.167 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.167 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:33.167 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.534 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:34.536 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:35.537 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:35.975 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:59:36.539 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:37.168 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:37.169 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:37.169 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:37.170 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:37.170 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:37.170 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:37.170 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:37.170 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:37.170 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:37.170 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:37.170 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:37.170 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:37.171 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:37.171 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:37.540 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:38.541 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:39.543 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:40.545 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:41.171 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:41.172 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:41.172 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:41.172 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:41.173 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:41.173 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:41.173 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:41.173 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:41.173 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:41.173 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:41.173 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:41.173 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:41.173 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:41.173 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:41.545 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:42.547 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:43.549 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:44.550 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:45.173 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:45.174 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:45.174 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:45.175 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:45.175 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:45.175 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:45.175 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:45.175 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:45.175 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.176 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:45.176 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.176 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.176 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:45.176 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.552 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:46.553 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:47.554 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:48.010 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:59:48.556 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:48.885 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:59:48.885 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:48.885 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:59:48.885 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.175 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:49.177 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:49.177 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:49.177 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:49.177 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:49.177 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:49.177 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:49.177 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:49.177 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.178 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:49.178 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.178 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.178 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:49.178 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.557 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:50.558 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:51.560 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:52.561 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:53.178 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:53.179 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:53.179 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:53.179 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:53.179 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:53.179 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:53.179 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:53.180 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:53.180 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.180 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:53.180 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.180 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.180 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:53.180 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.562 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:54.564 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:55.565 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:56.566 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:57.180 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:57.181 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:57.182 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:57.182 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:57.182 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:57.182 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:57.182 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:57.182 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:57.183 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.183 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:57.183 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.183 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.183 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:57.183 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.568 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:58.569 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:59.571 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:00.163 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:00:00.572 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:01.183 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:01.184 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:01.184 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:01.184 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:01.184 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:01.184 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:01.184 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:01.185 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:01.185 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.185 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:01.185 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.185 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.185 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:01.185 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.574 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:02.575 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:03.576 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:04.577 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:05.185 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:05.186 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:05.186 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:05.186 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:05.186 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:05.187 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:05.187 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:05.187 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:05.187 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:05.187 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:05.187 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:05.187 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:05.187 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:05.187 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:05.578 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:06.580 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:07.581 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:08.583 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:09.187 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:09.189 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:09.189 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:09.189 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:09.189 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:09.189 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:09.189 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:09.189 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:09.190 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:09.190 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:09.190 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:09.190 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:09.190 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:09.190 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:09.585 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:10.586 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:11.587 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:12.296 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:00:12.588 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:13.190 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:13.191 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:13.191 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:13.191 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:13.191 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:13.191 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:13.191 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:13.191 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:13.192 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.192 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:13.192 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.192 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.192 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:13.192 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.589 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:14.591 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:15.592 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:16.594 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:17.192 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:17.193 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:17.193 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:17.193 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:17.193 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:17.193 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:17.193 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:17.194 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:17.194 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.194 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:17.194 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.194 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.194 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:17.194 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.595 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:18.596 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:19.597 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:20.599 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:21.194 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:21.195 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:21.195 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:21.195 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:21.196 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:21.196 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:21.196 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:21.196 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:21.196 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.197 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:21.197 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.197 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.197 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:21.197 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.600 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:22.601 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:23.602 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:24.604 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:24.778 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:00:25.196 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:25.197 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:25.197 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:25.197 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:25.198 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:25.198 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:25.198 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:25.198 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:25.198 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.198 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:25.198 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.198 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.198 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:25.199 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.605 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:26.606 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:27.608 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:28.610 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:29.198 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:29.199 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:29.199 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:29.199 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:29.200 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:29.200 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:29.200 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:29.200 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:29.200 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:29.200 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:29.200 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:29.200 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:29.200 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:29.200 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:29.610 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:30.611 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:31.613 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:32.614 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:33.200 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:33.201 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:33.202 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:33.202 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:33.202 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:33.202 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:33.202 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:33.202 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:33.202 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:33.202 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:33.202 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:33.203 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:33.203 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:33.203 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:33.615 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:34.617 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:35.618 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:36.620 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:36.892 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:00:37.202 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:37.204 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:37.204 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:37.204 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:37.204 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:37.204 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:37.204 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:37.205 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:37.205 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:37.205 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:37.205 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:37.205 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:37.205 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:37.205 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:37.621 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:38.622 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:39.624 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:40.625 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:41.205 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:41.206 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:41.207 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:41.207 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:41.207 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:41.207 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:41.207 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:41.207 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:41.207 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:41.208 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:41.208 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:41.208 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:41.208 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:41.208 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:41.626 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:42.627 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:43.629 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:44.630 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:45.208 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:45.209 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:45.209 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:45.209 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:45.209 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:45.209 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:45.209 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:45.210 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:45.210 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.210 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:45.210 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.210 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.210 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:45.210 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.632 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:46.632 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:47.633 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:48.635 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:48.886 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:00:48.886 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:48.886 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:00:48.886 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:48.947 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:00:49.210 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:49.211 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:49.211 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:49.211 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:49.211 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:49.211 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:49.211 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:49.212 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:49.212 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.212 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:49.212 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.212 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.212 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:49.212 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.430 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) 09/01/2025 02:00:49.430 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) is either full or getting a new batch 09/01/2025 02:00:49.430 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:00:49.430 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:00:49.430 7fc2ad71ba40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) 09/01/2025 02:00:49.431 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:00:49.431 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x02\xc2\x9a\xd3\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x01n\x16\x00\x00\x01\x99\x03\x01n\x16\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:00:49.431 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x02\xc2\x9a\xd3\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x01n\x16\x00\x00\x01\x99\x03\x01n\x16\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:00:49.431 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x02\xc2\x9a\xd3\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x01n\x16\x00\x00\x01\x99\x03\x01n\x16\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:00:49.431 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 7 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x02\xc2\x9a\xd3\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x01n\x16\x00\x00\x01\x99\x03\x01n\x16\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:00:49.431 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 179931.927246, idle connection 299447.000000, request 29999.963284 09/01/2025 02:00:49.432 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991665, metadata 179931.406494, idle connection 299446.000000, request 29999.448299 09/01/2025 02:00:49.434 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 7 09/01/2025 02:00:49.434 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:00:49.434 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 7 (3.0202865600585938 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:00:49.435 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:00:49.435 7fc2ad644680 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) with base offset 0 log start offset 0. 09/01/2025 02:00:49.435 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993095, metadata 179928.463135, idle connection 299444.000000, request inf 09/01/2025 02:00:49.435 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:00:49.636 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:50.637 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:51.638 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:52.639 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:53.212 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:53.213 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:53.214 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:53.214 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:53.214 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:53.214 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:53.214 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:53.214 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:53.215 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.215 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:53.215 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.215 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.215 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:53.215 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.640 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:54.642 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:55.643 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:56.645 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:57.215 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:57.216 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:57.217 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:57.217 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:57.217 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:57.217 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:57.217 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:57.217 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:57.217 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.217 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:57.217 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.218 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.218 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:57.218 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.646 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:58.648 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:59.649 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:00.650 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:01.134 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:01:01.217 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:01.219 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:01.219 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:01.219 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:01.219 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:01.219 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:01.219 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:01.219 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:01.219 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.220 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:01.220 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.220 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.220 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:01.220 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.651 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:02.653 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:03.654 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:04.656 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:05.220 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:05.221 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:05.221 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:05.221 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:05.221 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:05.221 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:05.221 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:05.221 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:05.221 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.222 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:05.222 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.222 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.222 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:05.222 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.657 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:06.658 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:07.660 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:08.661 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:09.222 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:09.223 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:09.223 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:09.223 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:09.223 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:09.223 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:09.223 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:09.223 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:09.224 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.224 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:09.224 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.224 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.224 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:09.224 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.663 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:10.664 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:11.665 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:12.667 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:13.224 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:13.225 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:13.226 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:13.226 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:13.226 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:13.226 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:13.226 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:13.226 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:13.226 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:13.226 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:13.226 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:13.227 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:13.227 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:13.227 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:13.279 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:01:13.669 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:14.670 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:15.671 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:16.673 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:17.226 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:17.227 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:17.228 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:17.228 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:17.228 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:17.228 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:17.228 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:17.228 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:17.228 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.229 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:17.229 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.229 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.229 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:17.229 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.675 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:18.676 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:19.677 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:20.678 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:21.228 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:21.229 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:21.230 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:21.230 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:21.230 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:21.230 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:21.230 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:21.231 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:21.231 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.231 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:21.231 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.231 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.231 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:21.231 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.680 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:22.681 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:23.682 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:24.684 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:25.230 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:25.231 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:25.231 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:25.231 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:25.232 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:25.232 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:25.232 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:25.232 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:25.232 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.232 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:25.232 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.232 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.232 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:25.232 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.685 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:25.711 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:01:26.687 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:27.688 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:28.690 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:29.232 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:29.233 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:29.233 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:29.233 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:29.233 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:29.233 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:29.233 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:29.234 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:29.234 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.234 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:29.234 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.234 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.234 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:29.234 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.691 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:30.692 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:31.693 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:32.694 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:33.235 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:33.236 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:33.236 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:33.236 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:33.236 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:33.236 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:33.236 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:33.236 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:33.237 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.237 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:33.237 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.237 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.237 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:33.237 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.696 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:34.697 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:35.699 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:36.701 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:37.237 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:37.238 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:37.238 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:37.238 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:37.238 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:37.239 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:37.239 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:37.239 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:37.239 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:37.239 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:37.239 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:37.239 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:37.239 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:37.239 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:37.702 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:37.838 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:01:38.703 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:39.705 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:40.706 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:41.239 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:41.240 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:41.240 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:41.241 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:41.241 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:41.241 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:41.241 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:41.241 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:41.241 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:41.241 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:41.241 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:41.241 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:41.241 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:41.242 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:41.708 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:42.709 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:43.710 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:44.711 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:45.242 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:45.243 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:45.243 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:45.243 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:45.243 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:45.243 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:45.243 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:45.244 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:45.244 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.244 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:45.244 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.244 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.244 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:45.244 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.713 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:46.715 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:47.716 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:48.717 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:48.886 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:01:48.887 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:48.887 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:01:48.887 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.243 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:49.244 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:49.244 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:49.245 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:49.245 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:49.245 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:49.245 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:49.245 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:49.245 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.245 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:49.245 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.245 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.245 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:49.246 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.719 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:49.948 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:01:50.721 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:51.723 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:52.724 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:53.245 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:53.246 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:53.246 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:53.246 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:53.246 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:53.246 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:53.246 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:53.247 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:53.247 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.247 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:53.247 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.247 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.247 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:53.247 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.725 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:54.727 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:55.728 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:56.730 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:57.247 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:57.248 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:57.249 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:57.249 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:57.249 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:57.249 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:57.249 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:57.249 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:57.249 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.249 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:57.250 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.250 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.250 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:57.250 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.731 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:58.733 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:59.734 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:00.736 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:01.250 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:01.252 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:01.252 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:01.252 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:01.252 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:01.252 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:01.252 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:01.253 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:01.253 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.253 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:01.253 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.253 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.253 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:01.253 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.741 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:02.107 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:02.742 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:03.743 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:04.745 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:05.252 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:05.253 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:05.253 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:05.253 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:05.253 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:05.253 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:05.253 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:05.254 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:05.254 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.254 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:05.254 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.254 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.254 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:05.254 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.746 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:06.748 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:07.749 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:08.750 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:09.254 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:09.255 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:09.255 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:09.255 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:09.255 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:09.255 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:09.255 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:09.255 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:09.256 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:09.256 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:09.256 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:09.256 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:09.256 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:09.256 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:09.752 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:10.753 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:11.754 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:12.756 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:13.256 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:13.257 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:13.257 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:13.257 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:13.257 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:13.257 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:13.257 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:13.258 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:13.258 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:13.258 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:13.258 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:13.258 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:13.258 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:13.258 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:13.757 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:14.127 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:14.758 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:15.760 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:16.762 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:17.258 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:17.259 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:17.259 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:17.260 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:17.260 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:17.260 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:17.260 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:17.260 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:17.260 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.260 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:17.260 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.261 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.261 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:17.261 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.763 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:18.764 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:19.765 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:20.766 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:21.260 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:21.262 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:21.262 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:21.262 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:21.262 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:21.262 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:21.262 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:21.262 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:21.262 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.263 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:21.263 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.263 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.263 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:21.263 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.767 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:22.769 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:23.770 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:24.772 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:25.263 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:25.264 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:25.264 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:25.264 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:25.264 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:25.264 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:25.264 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:25.264 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:25.264 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.265 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:25.265 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.265 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.265 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:25.265 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.774 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:26.602 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:26.775 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:27.777 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:28.778 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:29.265 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:29.266 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:29.266 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:29.266 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:29.266 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:29.266 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:29.266 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:29.267 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:29.267 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.267 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:29.267 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.267 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.267 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:29.267 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.779 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:30.781 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:31.782 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:32.784 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:33.266 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:33.268 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:33.268 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:33.268 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:33.268 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:33.268 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:33.268 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:33.268 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:33.269 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.269 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:33.269 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.269 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.269 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:33.269 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.786 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:34.787 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:35.788 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:36.790 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:37.269 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:37.270 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:37.270 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:37.270 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:37.270 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:37.270 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:37.270 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:37.271 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:37.271 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:37.271 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:37.271 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:37.271 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:37.271 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:37.271 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:37.791 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:38.714 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:38.792 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:39.794 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:40.795 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:41.271 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:41.272 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:41.272 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:41.272 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:41.272 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:41.272 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:41.272 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:41.273 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:41.273 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:41.273 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:41.273 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:41.273 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:41.273 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:41.273 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:41.796 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:42.798 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:43.799 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:44.800 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:45.273 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:45.274 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:45.274 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:45.274 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:45.274 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:45.275 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:45.275 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:45.275 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:45.275 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.275 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:45.275 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.275 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.275 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:45.275 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.801 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:46.802 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:47.803 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:48.805 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:48.887 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:02:48.887 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:48.887 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:02:48.887 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.274 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:49.315 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:49.315 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:49.315 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:49.316 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:49.316 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:49.316 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:49.316 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:49.316 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.316 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:49.316 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.316 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.316 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:49.316 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.436 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=24) 09/01/2025 02:02:49.436 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=24) is either full or getting a new batch 09/01/2025 02:02:49.436 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:02:49.436 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:02:49.436 7fc2ad71ba40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=24) 09/01/2025 02:02:49.437 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:02:49.437 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=24, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02#\xc3\xdf\x83\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x03B\xdc\x00\x00\x01\x99\x03\x03B\xdc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:02:49.437 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=24, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02#\xc3\xdf\x83\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x03B\xdc\x00\x00\x01\x99\x03\x03B\xdc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:02:49.437 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=24, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02#\xc3\xdf\x83\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x03B\xdc\x00\x00\x01\x99\x03\x03B\xdc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:02:49.437 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 8 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=24, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02#\xc3\xdf\x83\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x03B\xdc\x00\x00\x01\x99\x03\x03B\xdc\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:02:49.438 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 59925.693115, idle connection 179441.000000, request 29999.961853 09/01/2025 02:02:49.438 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993572, metadata 59925.255371, idle connection 179440.000000, request 29999.531031 09/01/2025 02:02:49.441 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 8 09/01/2025 02:02:49.441 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:02:49.441 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 8 (3.2377243041992188 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=24, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:02:49.441 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=24, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:02:49.441 7fc2ad644680 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=24) with base offset 0 log start offset 0. 09/01/2025 02:02:49.441 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992142, metadata 59922.032471, idle connection 179437.000000, request inf 09/01/2025 02:02:49.441 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:02:49.806 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:50.736 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:50.807 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:51.809 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:52.811 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:53.316 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:53.317 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:53.317 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:53.317 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:53.317 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:53.317 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:53.317 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:53.318 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:53.318 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.318 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:53.318 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.318 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.318 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:53.318 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.812 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:54.814 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:55.815 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:56.816 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:57.317 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:57.319 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:57.319 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:57.319 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:57.319 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:57.319 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:57.319 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:57.319 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:57.319 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.320 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:57.320 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.320 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.320 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:57.320 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.818 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:58.819 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:59.820 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:00.822 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:01.319 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:01.320 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:01.321 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:01.321 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:01.321 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:01.321 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:01.321 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:01.321 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:01.321 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.322 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:01.322 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.322 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.322 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:01.322 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.824 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:02.767 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:02.824 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:03.826 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:04.828 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:05.322 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:05.323 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:05.323 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:05.323 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:05.323 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:05.323 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:05.323 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:05.324 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:05.324 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:05.324 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:05.324 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:05.324 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:05.324 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:05.324 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:05.829 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:06.831 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:07.832 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:08.833 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:09.324 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:09.325 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:09.325 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:09.325 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:09.325 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:09.325 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:09.325 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:09.326 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:09.326 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:09.326 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:09.326 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:09.326 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:09.326 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:09.326 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:09.835 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:10.836 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:11.837 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:12.839 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:13.326 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:13.327 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:13.327 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:13.328 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:13.328 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:13.328 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:13.328 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:13.328 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:13.328 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:13.328 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:13.328 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:13.328 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:13.329 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:13.329 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:13.840 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:14.841 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:14.888 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:15.844 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:16.845 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:17.328 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:17.330 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:17.330 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:17.330 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:17.330 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:17.330 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:17.330 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:17.330 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:17.330 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.331 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:17.331 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.331 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.331 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:17.331 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.847 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:18.849 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:19.851 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:20.852 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:21.331 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:21.332 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:21.332 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:21.332 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:21.332 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:21.332 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:21.332 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:21.333 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:21.333 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.333 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:21.333 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.333 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.333 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:21.333 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.853 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:22.855 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:23.855 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:24.857 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:25.333 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:25.334 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:25.334 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:25.334 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:25.334 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:25.334 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:25.334 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:25.335 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:25.335 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.335 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:25.335 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.335 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.335 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:25.335 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.859 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:26.861 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:27.402 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:27.863 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:28.864 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:29.335 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:29.336 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:29.336 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:29.336 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:29.337 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:29.337 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:29.337 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:29.337 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:29.337 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.337 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:29.338 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.338 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.338 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:29.338 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.866 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:30.867 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:31.869 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:32.870 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:33.338 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:33.339 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:33.339 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:33.339 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:33.339 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:33.339 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:33.339 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:33.340 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:33.340 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:33.340 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:33.340 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:33.340 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:33.340 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:33.340 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:33.872 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:34.874 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:35.875 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:36.876 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:37.340 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:37.341 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:37.342 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:37.342 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:37.342 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:37.342 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:37.342 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:37.342 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:37.342 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:37.342 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:37.343 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:37.343 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:37.343 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:37.343 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:37.877 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:38.879 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:39.549 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:39.880 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:40.882 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:41.343 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:41.344 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:41.344 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:41.344 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:41.345 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:41.345 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:41.345 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:41.345 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:41.345 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.345 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:41.345 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.345 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.345 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:41.346 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.884 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:42.885 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:43.887 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:44.888 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:45.345 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:45.346 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:45.346 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:45.346 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:45.346 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:45.346 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:45.346 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:45.347 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:45.347 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.347 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:45.347 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.347 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.347 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:45.347 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.889 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:46.890 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:47.891 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:48.888 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:03:48.889 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:48.889 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:03:48.889 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:48.892 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:49.347 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:49.348 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:49.348 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:49.348 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:49.348 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:49.348 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:49.348 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:49.349 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:49.349 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.349 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:49.349 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.349 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.349 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:49.349 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.365 7fc2ad644680 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:03:49.365 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:03:49.365 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 9 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:03:49.365 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.471912, metadata 30000.000000, idle connection 119513.000000, request 29999.973297 09/01/2025 02:03:49.365 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992619, metadata 30000.000000, idle connection 119513.000000, request 29999.542952 09/01/2025 02:03:49.367 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 9 09/01/2025 02:03:49.367 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:03:49.368 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 9 (2.908468246459961 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.33', port=9092, rack=None)], cluster_id='hYIJrOiPQM-832Z-Zin5Xw', controller_id=1, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=5, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=10, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=20, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=25, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=26, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=11, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=4, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=16, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=17, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=23, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=28, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=13, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=22, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=8, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=2, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=29, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=19, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=14, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=1, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=7, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[])])]) 09/01/2025 02:03:49.369 7fc2ad644680 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 1, groups: 0) 09/01/2025 02:03:49.369 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992619, metadata 299999.863037, idle connection 119509.000000, request inf 09/01/2025 02:03:49.894 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:50.895 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:51.646 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:51.896 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:52.897 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:53.348 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:53.350 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:53.350 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:53.350 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:53.350 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:53.350 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:53.350 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:53.351 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:53.351 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.351 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:53.351 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.351 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.351 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:53.351 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.899 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:54.901 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:55.902 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:56.903 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:57.350 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:57.351 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:57.352 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:57.352 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:57.352 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:57.352 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:57.352 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:57.352 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:57.352 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.352 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:57.353 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.353 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.353 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:57.353 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.905 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:58.906 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:59.907 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:00.908 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:01.353 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:01.354 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:01.354 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:01.354 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:01.354 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:01.354 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:01.354 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:01.355 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:01.355 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.355 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:01.355 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.355 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.355 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:01.355 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.910 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:02.911 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:03.782 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:03.912 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:04.913 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:05.354 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:05.355 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:05.356 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:05.356 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:05.356 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:05.356 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:05.356 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:05.356 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:05.356 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:05.356 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:05.357 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:05.357 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:05.357 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:05.357 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:05.915 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:06.916 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:07.917 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:08.918 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:09.357 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:09.358 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:09.358 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:09.358 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:09.358 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:09.358 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:09.358 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:09.359 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:09.359 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:09.359 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:09.359 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:09.359 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:09.359 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:09.359 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:09.920 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:10.921 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:11.922 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:12.924 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:13.359 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:13.360 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:13.360 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:13.360 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:13.360 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:13.360 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:13.360 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:13.361 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:13.361 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:13.361 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:13.361 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:13.361 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:13.361 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:13.361 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:13.925 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:14.926 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:15.874 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:15.927 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:16.928 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:17.361 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:17.362 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:17.363 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:17.363 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:17.363 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:17.363 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:17.363 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:17.363 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:17.363 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.363 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:17.364 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.364 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.364 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:17.364 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.930 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:18.931 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:19.933 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:20.935 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:21.363 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:21.364 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:21.365 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:21.365 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:21.365 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:21.365 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:21.365 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:21.365 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:21.365 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.365 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:21.365 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.366 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.366 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:21.366 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.937 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:22.939 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:23.941 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:24.942 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:25.365 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:25.366 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:25.367 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:25.367 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:25.367 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:25.367 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:25.367 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:25.367 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:25.367 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.368 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:25.368 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.368 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.368 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:25.368 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.944 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:26.944 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:27.998 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:28.320 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:28.999 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:29.367 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:29.368 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:29.368 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:29.369 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:29.369 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:29.369 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:29.369 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:29.369 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:29.369 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.369 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:29.369 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.370 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.370 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:29.370 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.001 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:31.002 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:32.004 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:33.005 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:33.369 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:33.371 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:33.371 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:33.371 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:33.371 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:33.371 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:33.371 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:33.371 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:33.372 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:33.372 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:33.372 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:33.372 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:33.372 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:33.372 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.007 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:35.009 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:36.010 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:37.012 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:37.371 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:37.372 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:37.373 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:37.373 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:37.373 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:37.373 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:37.373 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:37.373 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:37.373 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:37.373 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:37.374 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:37.374 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:37.374 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:37.374 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.015 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:39.016 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:40.018 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:40.455 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:41.020 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:41.373 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:41.374 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:41.374 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:41.374 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:41.375 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:41.375 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:41.375 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:41.375 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:41.375 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:41.375 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:41.375 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:41.375 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:41.375 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:41.375 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.022 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:43.023 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:44.025 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:45.027 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:45.375 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:45.376 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:45.376 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:45.377 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:45.377 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:45.377 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:45.377 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:45.377 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:45.378 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:45.378 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:45.378 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:45.378 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:45.378 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:45.378 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.028 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:47.029 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:48.031 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:48.890 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:04:48.890 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:48.890 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:04:48.890 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.032 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:49.377 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:49.378 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:49.379 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:49.379 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:49.379 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:49.379 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:49.379 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:49.379 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:49.379 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.379 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:49.380 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.380 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.380 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:49.380 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.442 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=23) 09/01/2025 02:04:49.442 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=23) is either full or getting a new batch 09/01/2025 02:04:49.443 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:04:49.443 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:04:49.443 7fc2ad71ba40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=23) 09/01/2025 02:04:49.443 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:04:49.443 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=23, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8d\x88\xd8\xbf\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x05\x17\xa2\x00\x00\x01\x99\x03\x05\x17\xa2\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:04:49.443 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=23, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8d\x88\xd8\xbf\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x05\x17\xa2\x00\x00\x01\x99\x03\x05\x17\xa2\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:04:49.444 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=23, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8d\x88\xd8\xbf\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x05\x17\xa2\x00\x00\x01\x99\x03\x05\x17\xa2\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:04:49.444 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 10 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=23, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8d\x88\xd8\xbf\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x05\x17\xa2\x00\x00\x01\x99\x03\x05\x17\xa2\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:04:49.444 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 239925.101318, idle connection 59434.000000, request 29999.962330 09/01/2025 02:04:49.444 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993334, metadata 239924.635010, idle connection 59434.000000, request 29999.503613 09/01/2025 02:04:49.447 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 10 09/01/2025 02:04:49.447 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:04:49.447 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 10 (3.077268600463867 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=23, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:04:49.447 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=23, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:04:49.447 7fc2ad644680 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=23) with base offset 0 log start offset 0. 09/01/2025 02:04:49.447 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993334, metadata 239921.616943, idle connection 59431.000000, request inf 09/01/2025 02:04:49.448 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:04:50.033 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:51.035 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:52.037 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:52.578 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:53.039 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:53.380 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:53.381 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:53.381 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:53.381 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:53.381 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:53.381 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:53.381 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:53.382 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:53.382 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.382 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:53.382 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.382 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.382 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:53.382 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.040 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:55.042 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:56.044 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:57.046 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:57.382 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:57.383 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:57.383 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:57.383 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:57.383 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:57.383 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:57.383 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:57.384 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:57.384 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:57.384 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:57.384 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:57.384 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:57.384 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:57.384 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.047 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:59.048 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:00.049 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:01.051 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:01.384 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:01.385 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:01.385 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:01.385 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:01.385 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:01.385 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:01.385 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:01.386 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:01.386 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:01.386 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:01.386 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:01.386 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:01.386 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:01.386 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.053 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:03.055 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:04.056 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:04.626 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:05.057 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:05.385 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:05.386 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:05.387 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:05.387 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:05.387 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:05.387 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:05.387 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:05.387 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:05.387 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:05.387 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:05.388 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:05.388 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:05.388 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:05.388 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.059 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:07.060 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:08.062 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:09.063 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:09.387 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:09.388 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:09.389 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:09.389 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:09.389 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:09.389 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:09.389 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:09.389 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:09.389 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:09.389 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:09.389 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:09.390 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:09.390 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:09.390 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.065 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:11.066 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:12.067 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:13.069 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:13.389 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:13.390 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:13.390 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:13.391 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:13.391 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:13.391 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:13.391 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:13.391 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:13.391 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:13.391 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:13.392 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:13.392 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:13.392 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:13.392 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.071 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:15.072 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:16.073 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:16.731 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:17.074 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:17.391 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:17.392 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:17.392 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:17.392 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:17.393 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:17.393 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:17.393 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:17.393 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:17.393 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.393 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:17.393 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.393 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.393 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:17.393 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.076 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:19.077 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:20.078 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:21.080 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:21.393 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:21.394 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:21.394 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:21.394 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:21.395 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:21.395 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:21.395 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:21.395 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:21.395 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:21.395 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:21.395 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:21.395 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:21.395 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:21.395 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.082 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:23.084 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:24.085 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:25.086 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:25.396 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:25.397 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:25.397 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:25.397 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:25.397 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:25.397 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:25.398 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:25.398 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:25.398 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.398 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:25.398 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.398 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.398 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:25.398 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.088 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:27.089 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:28.091 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:29.092 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:29.152 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:29.398 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:29.399 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:29.400 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:29.400 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:29.400 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:29.400 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:29.400 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:29.400 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:29.400 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.400 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:29.401 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.401 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.401 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:29.401 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.093 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:31.095 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:32.096 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:33.098 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:33.400 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:33.401 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:33.401 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:33.401 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:33.402 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:33.402 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:33.402 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:33.402 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:33.402 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:33.402 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:33.402 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:33.402 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:33.402 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:33.402 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.099 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:35.100 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:36.102 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:37.104 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:37.402 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:37.403 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:37.403 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:37.403 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:37.403 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:37.403 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:37.403 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:37.404 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:37.404 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:37.404 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:37.404 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:37.404 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:37.404 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:37.404 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.105 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:39.106 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:40.108 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:41.109 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:41.127 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:41.404 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:41.405 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:41.405 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:41.405 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:41.405 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:41.405 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:41.405 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:41.406 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:41.406 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:41.406 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:41.406 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:41.406 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:41.406 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:41.406 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.110 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:43.112 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:44.113 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:45.114 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:45.405 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:45.406 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:45.407 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:45.407 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:45.407 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:45.407 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:45.407 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:45.407 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:45.407 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:45.407 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:45.407 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:45.407 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:45.408 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:45.408 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.116 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:47.118 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:48.120 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:48.880 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.988089, metadata 180489.145264, idle connection 480566.000000, request inf 09/01/2025 02:05:48.890 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:05:48.891 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:48.891 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:05:48.891 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.121 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:49.408 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:49.409 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:49.409 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:49.409 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:49.409 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:49.409 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:49.409 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:49.409 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:49.410 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.410 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:49.410 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.410 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.410 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:49.410 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.123 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:51.125 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:52.126 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:53.128 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:53.205 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:53.409 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:53.411 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:53.411 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:53.411 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:53.411 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:53.411 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:53.411 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:53.412 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:53.412 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.412 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:53.412 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.412 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.413 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:53.413 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.129 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:55.130 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:56.132 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:57.133 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:57.411 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:57.413 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:57.413 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:57.413 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:57.413 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:57.414 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:57.414 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:57.414 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:57.414 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:57.414 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:57.414 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:57.414 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:57.414 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:57.414 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.135 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:59.136 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:00.138 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:01.139 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:01.413 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:01.415 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:01.415 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:01.415 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:01.415 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:01.415 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:01.415 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:01.415 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:01.415 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:01.416 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:01.416 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:01.416 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:01.416 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:01.416 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.142 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:03.144 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:04.145 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:05.146 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:05.376 7fc2ad713d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:05.415 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:05.416 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:05.416 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:05.417 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:05.417 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:05.417 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:05.417 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:05.417 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:05.417 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:05.417 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:05.417 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:05.417 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:05.418 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:05.418 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.148 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:07.149 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:08.151 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:09.152 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:09.417 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:09.418 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:09.418 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:09.418 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:09.418 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:09.418 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:09.418 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:09.419 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:09.419 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:09.419 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:09.419 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:09.419 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:09.419 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:09.419 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.154 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:11.156 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:12.157 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:13.159 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:13.419 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:13.420 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:13.421 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:13.421 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:13.421 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:13.421 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:13.421 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:13.421 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:13.421 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:13.421 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:13.421 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:13.421 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:13.422 7fc2ad713d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:13.422 7fc2ad713d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.161 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:15.162 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:16.164 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:17.165 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:17.421 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:17.422 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:17.422 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:17.422 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:17.422 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:17.422 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:17.422 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:17.423 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:17.423 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.423 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:17.423 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.423 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.423 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:17.423 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.461 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:18.166 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:19.167 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:20.169 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:21.171 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:21.423 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:21.424 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:21.424 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:21.424 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:21.424 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:21.424 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:21.424 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:21.425 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:21.425 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:21.425 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:21.425 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:21.425 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:21.425 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:21.425 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.173 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:23.175 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:24.176 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:25.177 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:25.425 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:25.426 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:25.426 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:25.427 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:25.427 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:25.427 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:25.427 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:25.427 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:25.427 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:25.427 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:25.427 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:25.428 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:25.428 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:25.428 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.179 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:27.181 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:28.182 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:29.183 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:29.427 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:29.428 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:29.429 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:29.429 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:29.429 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:29.429 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:29.429 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:29.429 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:29.429 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.429 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:29.430 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.430 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.430 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:29.430 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.917 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:30.184 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:31.185 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:31.986 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:32.187 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:33.189 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:33.430 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:33.431 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:33.431 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:33.431 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:33.431 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:33.431 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:33.431 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:33.432 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:33.432 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:33.432 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:33.432 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:33.432 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:33.432 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:33.432 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.089 7fc2ad644540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:34.190 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:35.191 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:36.193 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:37.195 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:37.432 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:37.433 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:37.433 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:37.433 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:37.433 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:37.433 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:37.433 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:37.434 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:37.434 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:37.434 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:37.434 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:37.434 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:37.434 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:37.434 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.197 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:39.198 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:40.200 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:41.201 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:41.434 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:41.435 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:41.435 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:41.436 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:41.436 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:41.436 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:41.436 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:41.436 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:41.436 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:41.436 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:41.436 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:41.436 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:41.437 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:41.437 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.203 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:43.204 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:44.204 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:45.206 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:45.436 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:45.437 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:45.438 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:45.438 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:45.438 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:45.438 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:45.438 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:45.438 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:45.438 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:45.438 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:45.438 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:45.438 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:45.439 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:45.439 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.208 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:47.209 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:48.211 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:48.891 7fc2ad71b2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:06:48.892 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:48.892 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:06:48.892 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:49.212 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:49.438 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:49.439 7fc2ad644540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:49.439 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:49.439 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:49.439 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:49.439 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:49.439 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:49.440 7fc2ad644540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:49.440 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:49.440 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:49.440 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:49.440 7fc2ad644540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:49.440 7fc2ad644540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:49.440 7fc2ad644540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:49.448 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=7) 09/01/2025 02:06:49.449 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=7) is either full or getting a new batch 09/01/2025 02:06:49.449 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:06:49.449 7fc2ad71ba40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:06:49.449 7fc2ad71ba40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=7) 09/01/2025 02:06:49.449 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:06:49.450 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, records=b"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02'\nog\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06\xeci\x00\x00\x01\x99\x03\x06\xeci\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00")])])} 09/01/2025 02:06:49.450 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, records=b"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02'\nog\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06\xeci\x00\x00\x01\x99\x03\x06\xeci\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00")])]) 09/01/2025 02:06:49.450 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, records=b"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02'\nog\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06\xeci\x00\x00\x01\x99\x03\x06\xeci\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00")])]) 09/01/2025 02:06:49.450 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Request 11 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, records=b"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02'\nog\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06\xeci\x00\x00\x01\x99\x03\x06\xeci\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00")])]) 09/01/2025 02:06:49.450 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 119918.939941, idle connection 419996.000000, request 29999.963522 09/01/2025 02:06:49.451 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992380, metadata 119918.438232, idle connection 419996.000000, request 29999.466658 09/01/2025 02:06:49.453 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Received correlation id: 11 09/01/2025 02:06:49.453 7fc2ad644680 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:06:49.453 7fc2ad644680 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.33', 9092)]>: Response 11 (2.722024917602539 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:06:49.453 7fc2ad644680 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=7, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:06:49.453 7fc2ad644680 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=7) with base offset 0 log start offset 0. 09/01/2025 02:06:49.453 7fc2ad644680 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990950, metadata 119915.721680, idle connection 419993.000000, request inf 09/01/2025 02:06:49.453 7fc2ad71ba40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:06:50.213 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:51.215 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:52.217 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:53.218 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:53.439 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:53.441 7fc2ad644360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:53.441 7fc2ad644360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:53.441 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:53.441 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:53.441 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:53.441 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:53.442 7fc2ad644360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:53.442 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:53.442 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:53.442 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:53.442 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:53.443 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:53.443 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.219 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:55.221 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:56.222 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:57.224 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:57.442 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:57.443 7fc2ad644360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:57.443 7fc2ad644360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:57.443 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:57.443 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:57.443 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:57.443 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:57.443 7fc2ad644360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:57.444 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:57.444 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:57.444 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:57.444 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:57.444 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:57.444 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.226 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:59.227 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:00.229 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:00.308 7fc2ad644360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:01.231 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:01.444 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:01.445 7fc2ad644360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:01.445 7fc2ad644360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:01.446 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:01.446 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:01.446 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:01.446 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:01.446 7fc2ad644360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:01.446 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:01.446 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:01.446 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:01.446 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:01.446 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:01.447 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.232 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:03.233 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:04.234 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:05.236 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:05.446 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:05.447 7fc2ad644360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:05.447 7fc2ad644360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:05.447 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:05.447 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:05.447 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:05.447 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:05.447 7fc2ad644360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:05.447 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:05.448 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.33:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.33:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.33:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.33:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:05.448 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:05.448 7fc2ad644360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:05.448 7fc2ad644360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3994-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691508874423 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:05.448 7fc2ad644360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.237 7fc2ad71b540 [contrail-alarm-gen] [INFO]: UVE Done