09/01/2025 01:59:14.683 7f1bc75da080 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:59:14.683 7f1bc75da080 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:59:14.683 7f1bc75da080 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:59:14.690 7f1bc75da080 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/01/2025 01:59:14.690 7f1bc3586d60 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/01/2025 01:59:14.690 7f1bc3586d60 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/01/2025 01:59:14.690 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:59:14.691 7f1bc35aac20 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:59:14.691 7f1bc35aac20 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:59:14.691 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:59:14.691 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.691 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:14.691 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.691 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.691 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/01/2025 01:59:14.692 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.692 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.692 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:59:14.692 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.692 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.692 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:59:14.693 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.693 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.693 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:59:14.693 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.693 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.693 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:59:14.693 7f1bc35aac20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.693 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:59:14.693 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/01/2025 01:59:14.694 7f1bc3552d60 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/01/2025 01:59:14.695 7f1bc34ecea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.695 7f1bc34ecea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:59:14.695 7f1bc34ecea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.695 7f1bc34ecea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.695 7f1bc34ecea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:59:14.695 7f1bc34ecea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.751 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.131:5673//'] 09/01/2025 01:59:14.758 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:59:14.759 7f1bc34e5d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.759 7f1bc34e5d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.759 7f1bc34e5d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.759 7f1bc34e5d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.760 7f1bc34e5d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.760 7f1bc34e5d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.760 7f1bc34ff720 [kazoo.client] [INFO]: Connecting to 10.0.0.131(10.0.0.131):2181, use_ssl: False 09/01/2025 01:59:14.761 7f1bc34ff720 [kazoo.client] [DEBUG]: Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=60000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) 09/01/2025 01:59:14.763 7f1bc34ff0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.763 7f1bc34ff0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.763 7f1bc34ff0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.765 7f1bc34ff720 [kazoo.client] [INFO]: Zookeeper connection established, state: CONNECTED 09/01/2025 01:59:14.765 7f1bc34ff720 [contrail-alarm-gen] [INFO]: Analytics Discovery listen CONNECTED 09/01/2025 01:59:14.765 7f1bc34ff720 [contrail-alarm-gen] [INFO]: Analytics Discovery to publish None 09/01/2025 01:59:14.766 7f1bc34ff0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.766 7f1bc34ff0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper re-established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.766 7f1bc34ff0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.766 7f1bc34ff0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.766 7f1bc34ff0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.766 7f1bc34ff0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.767 7f1bc34ff720 [kazoo.client] [DEBUG]: Sending request(xid=1): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:14.768 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:59:14.768 7f1bc34ff0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.768 7f1bc34ff0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.768 7f1bc34ff0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.768 7f1bc34ff0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.768 7f1bc34ff0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.131:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:59:14.768 7f1bc34ff0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.775 7f1bc3452220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.775 7f1bc3452220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:14.775 7f1bc3452220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.776 7f1bc34ff720 [kazoo.client] [DEBUG]: Received response(xid=1): ZnodeStat(czxid=4294967571, mzxid=4294967571, ctime=1756691938781, mtime=1756691938781, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=4294967571) 09/01/2025 01:59:14.776 7f1bc34ff720 [kazoo.client] [DEBUG]: Sending request(xid=2): GetChildren(path='/analytics-discovery-/AlarmGenerator', watcher=>) 09/01/2025 01:59:14.777 7f1bc34ff720 [kazoo.client] [DEBUG]: Received response(xid=2): [] 09/01/2025 01:59:14.777 7f1bc3552d60 [contrail-alarm-gen] [INFO]: Analytics Discovery Watcher AlarmGenerator Children [] 09/01/2025 01:59:14.777 7f1bc34ff720 [kazoo.client] [DEBUG]: Sending request(xid=3): Exists(path='/analytics-discovery-/Collector', watcher=None) 09/01/2025 01:59:14.778 7f1bc34ff720 [kazoo.client] [DEBUG]: Received response(xid=3): ZnodeStat(czxid=4294967572, mzxid=4294967572, ctime=1756691938785, mtime=1756691938785, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=4294967572) 09/01/2025 01:59:14.778 7f1bc34ff720 [kazoo.client] [DEBUG]: Sending request(xid=4): GetChildren(path='/analytics-discovery-/Collector', watcher=>) 09/01/2025 01:59:14.779 7f1bc34ff720 [kazoo.client] [DEBUG]: Received response(xid=4): [] 09/01/2025 01:59:14.780 7f1bc3552d60 [contrail-alarm-gen] [INFO]: Analytics Discovery AlarmGenerator reconnect 09/01/2025 01:59:14.780 7f1bc3552d60 [contrail-alarm-gen] [INFO]: ensure /analytics-discovery-/AlarmGenerator 09/01/2025 01:59:14.780 7f1bc3552d60 [contrail-alarm-gen] [INFO]: zk state CONNECTED (CONNECTED) 09/01/2025 01:59:14.780 7f1bc34ff720 [kazoo.client] [DEBUG]: Sending request(xid=5): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:14.781 7f1bc34ff720 [kazoo.client] [DEBUG]: Received response(xid=5): ZnodeStat(czxid=4294967571, mzxid=4294967571, ctime=1756691938781, mtime=1756691938781, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=4294967571) 09/01/2025 01:59:14.781 7f1bc3552d60 [contrail-alarm-gen] [INFO]: check for /analytics-discovery-/AlarmGenerator/cn-jenkins-deploy-platform-ansible-os-3995-3.-0 09/01/2025 01:59:14.781 7f1bc34ff720 [kazoo.client] [DEBUG]: Sending request(xid=6): Exists(path='/analytics-discovery-/AlarmGenerator/cn-jenkins-deploy-platform-ansible-os-3995-3.-0', watcher=None) 09/01/2025 01:59:14.783 7f1bc34ff720 [kazoo.client] [DEBUG]: Sending request(xid=7): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:14.783 7f1bc34ff720 [kazoo.client] [DEBUG]: Received response(xid=7): ZnodeStat(czxid=4294967571, mzxid=4294967571, ctime=1756691938781, mtime=1756691938781, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=4294967571) 09/01/2025 01:59:14.784 7f1bc34ff720 [kazoo.client] [DEBUG]: Sending request(xid=8): GetChildren(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:14.784 7f1bc34ff720 [kazoo.client] [DEBUG]: Received response(xid=8): [] 09/01/2025 01:59:14.785 7f1bc3552d60 [contrail-alarm-gen] [INFO]: Discovery AG callback : dict_values([]) 09/01/2025 01:59:14.827 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:59:14.827 7f1bc3452e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.828 7f1bc3452e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.828 7f1bc3452e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.830 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:59:14.831 7f1bc3452e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.831 7f1bc3452e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:14.831 7f1bc3452e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.831 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/01/2025 01:59:14.831 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/01/2025 01:59:14.832 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:59:14.832 7f1bc3452e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:59:14.833 7f1bc3401cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.833 7f1bc3401cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:14.833 7f1bc3401cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.833 7f1bc3401cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.833 7f1bc3401cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:14.833 7f1bc3401cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.837 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:59:14.850 7f1bc3452e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:59:14.850 7f1bc3416040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.850 7f1bc3416040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:14.850 7f1bc3416040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.874 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:59:14.875 7f1bc33c99a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.876 7f1bc33c99a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:59:14.876 7f1bc33c99a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.883 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:59:14.883 7f1bc33c9220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.884 7f1bc33c9220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:14.884 7f1bc33c9220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.889 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:59:14.905 7f1bc35aac20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:59:14.906 7f1bc2d12a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.906 7f1bc2d12a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:14.906 7f1bc2d12a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.973 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config config_type = global_system_config operation = UPDATE config = { uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"fq_name : ["default-global-system-config"]autonomous_system : 64512enable_4byte_as : falseconfig_version : "1.0"ibgp_auto_mesh : truemac_aging_time : 300igmp_enable : falseenable_security_policy_draft : falseid_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16062397226589833301, "uuid_lslong": 9453937659331017976}, "enable": true, "created": "2025-09-01T01:56:54.837560", "last_modified": "2025-09-01T01:57:57.527758", "description": null, "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"], "attr": null, "uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": null, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": null, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}] } 09/01/2025 01:59:14.973 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-address-mismatch-compute config_type = alarm operation = UPDATE config = { uuid : "fa3f6a2e-8282-4f6f-80ce-33639a52499d"fq_name : ["default-global-system-config", "system-defined-address-mismatch-compute"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_ip_address", "operand2": {"uve_attribute": "VrouterAgent.control_ip", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18032248181005373295, "uuid_lslong": 9281412384941033885}, "enable": true, "created": "2025-09-01T01:57:49.827148", "last_modified": "2025-09-01T01:57:50.038909", "description": "Compute Node IP Address mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-address-mismatch-compute" } 09/01/2025 01:59:14.973 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-address-mismatch-control config_type = alarm operation = UPDATE config = { uuid : "c8266300-cc83-4a49-bd56-9c8537aad9c1"fq_name : ["default-global-system-config", "system-defined-address-mismatch-control"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "not in", "operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14422323711782898249, "uuid_lslong": 13643264217144023489}, "enable": true, "created": "2025-09-01T01:57:49.990142", "last_modified": "2025-09-01T01:57:50.257402", "description": "Control Node IP Address mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-address-mismatch-control" } 09/01/2025 01:59:14.974 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-bgp-connectivity config_type = alarm operation = UPDATE config = { uuid : "81d55dd2-bdf1-491a-b083-a3fc512867ea"fq_name : ["default-global-system-config", "system-defined-bgp-connectivity"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "BgpRouterState.num_up_bgp_peer", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "BgpRouterState.num_up_bgp_peer", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9355486960643295514, "uuid_lslong": 12719190076713101290}, "enable": true, "created": "2025-09-01T01:57:50.146892", "last_modified": "2025-09-01T01:57:50.443758", "description": "BGP peer mismatch. Not enough BGP peers are up.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-bgp-connectivity" } 09/01/2025 01:59:14.974 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-bottle-request-size-limit config_type = alarm operation = UPDATE config = { uuid : "b9cad561-1c37-4571-bb68-f9d9eca108c7"fq_name : ["default-global-system-config", "system-defined-bottle-request-size-limit"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["config-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "UveConfigReq.err_info.*.", "operand2": {"uve_attribute": null, "json_value": "True"}, "variables": []}]}]}id_perms : {"permissions": null, "uuid": {"uuid_mslong": 13387747455374148977, "uuid_lslong": 13504318197043628231}, "enable": null, "created": "2025-09-01T01:56:56.131127", "last_modified": "2025-09-01T01:56:56.131127", "description": "Bottle request size limit exceeded.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []} } 09/01/2025 01:59:14.974 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-conf-elements-inconsistency config_type = alarm operation = UPDATE config = { uuid : "03bc6282-7e14-4f38-badd-7f9f072b4620"fq_name : ["default-global-system-config", "system-defined-conf-elements-inconsistency"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "size==", "operand1": "ContrailConfig.elements", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 269198390335917880, "uuid_lslong": 13465058781904455200}, "enable": true, "created": "2025-09-01T01:57:49.704740", "last_modified": "2025-09-01T01:57:49.826527", "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-conf-elements-inconsistency" } 09/01/2025 01:59:14.974 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-conf-incorrect config_type = alarm operation = UPDATE config = { uuid : "b17d423d-1f4f-42e4-ab55-b705b7d714e6"fq_name : ["default-global-system-config", "system-defined-conf-incorrect"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "ContrailConfig", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12789451347087803108, "uuid_lslong": 12345975188693128422}, "enable": true, "created": "2025-09-01T01:57:50.244269", "last_modified": "2025-09-01T01:57:50.614160", "description": "ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-conf-incorrect" } 09/01/2025 01:59:14.975 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-core-files config_type = alarm operation = UPDATE config = { uuid : "29f93096-c40c-408a-94a3-070ccbe9c499"fq_name : ["default-global-system-config", "system-defined-core-files"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "NodeStatus.all_core_file_list", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "NodeStatus.all_core_file_list", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3024502048848363658, "uuid_lslong": 10710412090359268505}, "enable": true, "created": "2025-09-01T01:57:51.398898", "last_modified": "2025-09-01T01:57:52.970710", "description": "A core file has been generated on the node.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-core-files" } 09/01/2025 01:59:14.975 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-disk-usage-critical config_type = alarm operation = UPDATE config = { uuid : "149fff35-9ae3-4f8d-906f-8423cd373cb2"fq_name : ["default-global-system-config", "system-defined-disk-usage-critical"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": ">", "operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operand2": {"uve_attribute": null, "json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1486187007752490893, "uuid_lslong": 10407682553178176690}, "enable": true, "created": "2025-09-01T01:57:50.437138", "last_modified": "2025-09-01T01:57:50.956619", "description": "Disk usage crosses critical threshold limit.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-disk-usage-critical" } 09/01/2025 01:59:14.975 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-disk-usage-high config_type = alarm operation = UPDATE config = { uuid : "04d7992e-6679-4bd3-8298-737ed5150d31"fq_name : ["default-global-system-config", "system-defined-disk-usage-high"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "range", "operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operand2": {"uve_attribute": null, "json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 348915920711273427, "uuid_lslong": 9410398409968651569}, "enable": true, "created": "2025-09-01T01:57:50.343229", "last_modified": "2025-09-01T01:57:50.782298", "description": "Disk usage crosses high threshold limit.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-disk-usage-high" } 09/01/2025 01:59:14.975 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-node-status config_type = alarm operation = UPDATE config = { uuid : "64598318-cd07-4a87-af8d-efa93bf77779"fq_name : ["default-global-system-config", "system-defined-node-status"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7230954819262302855, "uuid_lslong": 12650030438488176505}, "enable": true, "created": "2025-09-01T01:57:50.537284", "last_modified": "2025-09-01T01:57:51.124011", "description": "Node Failure. NodeStatus UVE not present.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-node-status" } 09/01/2025 01:59:14.975 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-package-version-mismatch config_type = alarm operation = UPDATE config = { uuid : "f72985ec-b0f3-47a5-94cf-753bc9704e58"fq_name : ["default-global-system-config", "system-defined-package-version-mismatch"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "NodeStatus.running_package_version", "operand2": {"uve_attribute": "NodeStatus.installed_package_version", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17809913453040846757, "uuid_lslong": 10722918137435541080}, "enable": true, "created": "2025-09-01T01:57:51.577756", "last_modified": "2025-09-01T01:57:53.321232", "description": "There is a mismatch between installed and running package version.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-package-version-mismatch" } 09/01/2025 01:59:14.976 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-partial-sysinfo config_type = alarm operation = UPDATE config = { uuid : "d7676398-52ea-482c-a5c5-2fc5683b53c4"fq_name : ["default-global-system-config", "system-defined-partial-sysinfo"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.build_info", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15521484146632968236, "uuid_lslong": 11945006111573889988}, "enable": true, "created": "2025-09-01T01:57:50.636982", "last_modified": "2025-09-01T01:57:51.305614", "description": "System Info Incomplete.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-partial-sysinfo" } 09/01/2025 01:59:14.976 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks config_type = alarm operation = UPDATE config = { uuid : "d08b3c4f-8364-42cc-b19f-c64aec697ac0"fq_name : ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["database-nodeconfig-database-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": ">=", "operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"uve_attribute": null, "json_value": "300"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15027170893856260812, "uuid_lslong": 12799166691106454208}, "enable": true, "created": "2025-09-01T01:57:51.495240", "last_modified": "2025-09-01T01:57:53.136268", "description": "Pending compaction tasks in cassandra crossed the configured threshold.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-pending-cassandra-compaction-tasks" } 09/01/2025 01:59:14.976 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-process-connectivity config_type = alarm operation = UPDATE config = { uuid : "6177a338-54f3-47b2-b8cd-742fd97fe3cd"fq_name : ["default-global-system-config", "system-defined-process-connectivity"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.process_status", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "NodeStatus.process_status.state", "operand2": {"uve_attribute": null, "json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7023261606246303666, "uuid_lslong": 13316427422065746893}, "enable": true, "created": "2025-09-01T01:57:50.713917", "last_modified": "2025-09-01T01:57:51.470711", "description": "Process(es) reporting as non-functional.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-process-connectivity" } 09/01/2025 01:59:14.976 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-process-status config_type = alarm operation = UPDATE config = { uuid : "810dc9aa-8015-43ae-903f-6caad1a28708"fq_name : ["default-global-system-config", "system-defined-process-status"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.process_info", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "NodeStatus.process_info.process_state", "operand2": {"uve_attribute": null, "json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9299310539720442798, "uuid_lslong": 10394145945911723784}, "enable": true, "created": "2025-09-01T01:57:50.802208", "last_modified": "2025-09-01T01:57:51.647167", "description": "Process Failure.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-process-status" } 09/01/2025 01:59:14.977 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-prouter-connectivity config_type = alarm operation = UPDATE config = { uuid : "2b30d6f6-0211-4eb5-99c7-79101d9a1ff9"fq_name : ["default-global-system-config", "system-defined-prouter-connectivity"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["prouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.connected_agent_list", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3112223694598000309, "uuid_lslong": 11080958518291472377}, "enable": true, "created": "2025-09-01T01:57:50.885227", "last_modified": "2025-09-01T01:57:51.816652", "description": "Prouter connectivity to controlling tor agent does not exist we look for non-empty value for connected_agent_list", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-prouter-connectivity" } 09/01/2025 01:59:14.977 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-prouter-tsn-connectivity config_type = alarm operation = UPDATE config = { uuid : "935eb4db-73bc-4a29-8c4d-e2ed2941efd3"fq_name : ["default-global-system-config", "system-defined-prouter-tsn-connectivity"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["prouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.tsn_agent_list", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}, {"operation": "!=", "operand1": "ProuterData.gateway_mode", "operand2": {"uve_attribute": null, "json_value": "\"SERVER\""}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.tsn_agent_list", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": []}, {"operation": "==", "operand1": "ProuterData.gateway_mode", "operand2": {"uve_attribute": null, "json_value": "\"SERVER\""}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10619123826018765353, "uuid_lslong": 10109986246743945171}, "enable": true, "created": "2025-09-01T01:57:50.981229", "last_modified": "2025-09-01T01:57:51.990045", "description": "Prouter connectivity to controlling tsn agent does not exist we look for non-empty value for tsn_agent_list", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-prouter-tsn-connectivity" } 09/01/2025 01:59:14.977 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-storage-cluster-state config_type = alarm operation = UPDATE config = { uuid : "8e1480d5-277f-405c-a1ae-7e859515d832"fq_name : ["default-global-system-config", "system-defined-storage-cluster-state"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["storage-cluster"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "StorageCluster.info_stats.status", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10237949505899020380, "uuid_lslong": 11650388398251038770}, "enable": true, "created": "2025-09-01T01:57:51.062109", "last_modified": "2025-09-01T01:57:52.195183", "description": "Storage Cluster warning/errors.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-storage-cluster-state" } 09/01/2025 01:59:14.977 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-interface config_type = alarm operation = UPDATE config = { uuid : "a3cde80b-b0cf-4715-9c86-8fec97c15a7b"fq_name : ["default-global-system-config", "system-defined-vrouter-interface"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": ">=", "operand1": "VrouterAgent.down_interface_count", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11803345335316596501, "uuid_lslong": 11278860563117070971}, "enable": true, "created": "2025-09-01T01:57:51.143961", "last_modified": "2025-09-01T01:57:52.395443", "description": "Vrouter interface(s) down.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-interface" } 09/01/2025 01:59:14.977 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down config_type = alarm operation = UPDATE config = { uuid : "0611e1ec-1ca8-48db-9ad2-92f299cb8e5c"fq_name : ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.l3mh_phy_interface_down", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 437379043040970971, "uuid_lslong": 11156140797610135132}, "enable": true, "created": "2025-09-01T01:57:51.847772", "last_modified": "2025-09-01T01:57:53.866729", "description": "On l3 multihomed compute one or more physical interfaces are Inactive.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-l3mh-physical-interface-link-down" } 09/01/2025 01:59:14.978 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-limit-exceeded config_type = alarm operation = UPDATE config = { uuid : "af6de2d9-8690-4972-9338-5b61b5922f04"fq_name : ["default-global-system-config", "system-defined-vrouter-limit-exceeded"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.res_limit", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12641009152992233842, "uuid_lslong": 10608329397487415044}, "enable": true, "created": "2025-09-01T01:57:51.666562", "last_modified": "2025-09-01T01:57:53.502926", "description": "Agent resource usage exceeded configured watermark for resource.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-limit-exceeded" } 09/01/2025 01:59:14.978 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-table-limit-exceeded config_type = alarm operation = UPDATE config = { uuid : "ca0e784a-4398-4836-afc4-17fc122957e4"fq_name : ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.res_table_limit", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14559706905692358710, "uuid_lslong": 12665274423476574180}, "enable": true, "created": "2025-09-01T01:57:51.751034", "last_modified": "2025-09-01T01:57:53.679612", "description": "Agent resource usage exceeded table size for resource in vrouter.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-table-limit-exceeded" } 09/01/2025 01:59:14.978 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-xmpp-close-reason config_type = alarm operation = UPDATE config = { uuid : "8e0c7107-46a5-4e13-bb2e-4bc845f22ee7"fq_name : ["default-global-system-config", "system-defined-xmpp-close-reason"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["xmpp-peer"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "XmppPeerInfoData.close_reason", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "!=", "operand1": "XmppPeerInfoData.state_info.state", "operand2": {"uve_attribute": null, "json_value": "\"Established\""}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10235680329170243091, "uuid_lslong": 13487801257560256231}, "enable": true, "created": "2025-09-01T01:57:51.312606", "last_modified": "2025-09-01T01:57:52.796846", "description": "XMPP connection closed towards peer, alarm has close reason", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-xmpp-close-reason" } 09/01/2025 01:59:14.978 7f1bc35aac20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-xmpp-connectivity config_type = alarm operation = UPDATE config = { uuid : "c634c7c0-0322-4d42-9aa5-5929ed1b749f"fq_name : ["default-global-system-config", "system-defined-xmpp-connectivity"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "BgpRouterState.num_up_xmpp_peer", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "BgpRouterState.num_up_xmpp_peer", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14282259945798913346, "uuid_lslong": 11143410889604691103}, "enable": true, "created": "2025-09-01T01:57:51.226178", "last_modified": "2025-09-01T01:57:52.608543", "description": "XMPP peer mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-xmpp-connectivity" } 09/01/2025 01:59:15.695 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:16.697 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:17.699 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:18.693 7f1bc3401900 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:18.695 7f1bc3401900 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:18.695 7f1bc3401900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:18.696 7f1bc3401900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:18.696 7f1bc3401900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:18.696 7f1bc3401900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:18.696 7f1bc3401900 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:18.697 7f1bc3401900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:18.697 7f1bc3401900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.697 7f1bc3401900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:18.697 7f1bc3401900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.697 7f1bc3401900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.697 7f1bc3401900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:18.698 7f1bc3401900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.700 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:19.701 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:19.770 7f1bc3401900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.771 7f1bc3401900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:19.771 7f1bc3401900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.771 7f1bc3401900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.771 7f1bc3401900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:19.771 7f1bc3401900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.771 7f1bc3401900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.771 7f1bc3401900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:19.771 7f1bc3401900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:20.703 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:21.705 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:21.873 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:59:21.867078', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:21.867078', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13422430401004267031, 'uuid_mslong': 671617895336398686}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '095210ea-dc52-4f5e-ba46-0d55aa436617'}, 'oper': 'CREATE', 'request-id': 'req-0bcb5258-1518-4af3-bc32-66c2afda7f6a', 'type': 'analytics_snmp_node', 'uuid': '095210ea-dc52-4f5e-ba46-0d55aa436617'} 09/01/2025 01:59:21.873 7f1bc3401540 [contrail-alarm-gen] [ERROR]: Error in rabbitmq drainer greenlet: 'AnalyticsLogger' object has no attribute 'warn' 09/01/2025 01:59:21.873 7f1bc3401540 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:59:21.874 7f1bc3401900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.874 7f1bc3401900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:21.874 7f1bc3401900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.878 7f1bc3401540 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:59:21.878 7f1bc3416220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.878 7f1bc3416220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:21.878 7f1bc3416220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.232 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:59:22.227040', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:22.227040', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10043001507123339914, 'uuid_mslong': 13054484782216790713}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b52ad8be-69c6-42b9-8b5f-e8a91ecca28a'}, 'oper': 'CREATE', 'request-id': 'req-cc52ae74-0d4d-419c-a865-1eba05a95d49', 'type': 'analytics_snmp_node', 'uuid': 'b52ad8be-69c6-42b9-8b5f-e8a91ecca28a'} 09/01/2025 01:59:22.233 7f1bc3401540 [contrail-alarm-gen] [ERROR]: Error in rabbitmq drainer greenlet: 'AnalyticsLogger' object has no attribute 'warn' 09/01/2025 01:59:22.233 7f1bc3401540 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:59:22.233 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.233 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:22.233 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.237 7f1bc3401540 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:59:22.237 7f1bc2d12e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.237 7f1bc2d12e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:22.237 7f1bc2d12e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.307 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:59:22.300301', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:22.300301', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10295412184530401449, 'uuid_mslong': 11790706739729089604}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a3a1014f-585b-4444-8ee0-a6d59b29b8a9'}, 'oper': 'CREATE', 'request-id': 'req-79a7348d-975b-4e66-ac0e-d0c0e9ecf4b1', 'type': 'analytics_snmp_node', 'uuid': 'a3a1014f-585b-4444-8ee0-a6d59b29b8a9'} 09/01/2025 01:59:22.307 7f1bc3401540 [contrail-alarm-gen] [ERROR]: Error in rabbitmq drainer greenlet: 'AnalyticsLogger' object has no attribute 'warn' 09/01/2025 01:59:22.307 7f1bc3401540 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:59:22.308 7f1bc2d12e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.308 7f1bc2d12e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:22.308 7f1bc2d12e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.313 7f1bc3401540 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:59:22.314 7f1bc2cc0540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.314 7f1bc2cc0540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:22.314 7f1bc2cc0540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.695 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:22.696 7f1bc2cc0400 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:22.697 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:22.697 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:22.697 7f1bc2cc0400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:22.697 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:22.697 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:22.698 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:22.698 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.698 7f1bc2cc0400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:22.698 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.698 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.698 7f1bc2cc0400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:22.698 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.707 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:23.708 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:24.709 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:25.711 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:26.698 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:26.699 7f1bc2cc0400 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:26.699 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:26.699 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:26.699 7f1bc2cc0400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:26.699 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:26.699 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:26.699 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:26.699 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.699 7f1bc2cc0400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:26.699 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.700 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.700 7f1bc2cc0400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:26.700 7f1bc2cc0400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.712 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:27.713 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:28.714 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:29.716 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:30.701 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:30.702 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:30.702 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:30.702 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:30.703 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:30.703 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:30.703 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:30.703 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:30.703 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.703 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:30.704 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.704 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.704 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:30.704 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.718 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:31.720 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:32.722 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:33.723 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:34.703 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:34.705 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:34.705 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:34.705 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:34.705 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:34.705 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:34.705 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:34.705 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:34.705 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.706 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:34.706 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.706 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.706 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:34.706 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.724 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:35.726 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:36.727 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:37.729 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:38.706 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:38.707 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:38.707 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:38.707 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:38.707 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:38.708 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:38.708 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:38.708 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:38.708 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.708 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:38.708 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.708 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.708 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:38.708 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.730 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:39.731 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:40.732 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:41.733 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:42.709 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:42.710 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:42.710 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:42.710 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:42.710 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:42.711 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:42.711 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:42.711 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:42.711 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.711 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:42.711 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.711 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.711 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:42.711 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.734 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:43.735 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:44.737 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:45.740 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:46.711 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:46.712 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:46.712 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:46.712 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:46.713 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:46.713 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:46.713 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:46.713 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:46.713 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.713 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:46.713 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.713 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.713 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:46.713 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.742 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:47.743 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:48.745 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:49.746 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:50.713 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:50.714 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:50.715 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:50.715 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:50.715 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:50.715 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:50.715 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:50.715 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:50.715 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.715 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:50.715 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.716 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.716 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:50.716 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.748 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:51.749 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:52.751 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:53.753 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:54.716 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:54.717 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:54.717 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:54.717 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:54.717 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:54.717 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:54.717 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:54.718 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:54.718 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.718 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:54.718 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.718 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.718 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:54.718 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.754 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:55.756 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:56.757 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:57.759 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:58.717 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:58.719 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:58.719 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:58.719 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:58.719 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:58.720 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:58.720 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:58.720 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:58.720 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.720 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:58.720 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.720 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.720 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:58.720 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.760 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:59.761 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:00.763 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:01.765 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:02.721 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:02.722 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:02.722 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:02.722 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:02.722 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:02.722 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:02.722 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:02.722 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:02.723 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.723 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:02.723 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.723 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.723 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:02.723 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.766 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:03.768 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:04.770 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:05.771 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:06.723 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:06.725 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:06.725 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:06.725 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:06.725 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:06.725 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:06.726 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:06.726 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:06.726 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.726 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:06.726 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.727 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.727 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:06.727 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.772 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:07.774 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:08.776 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:09.777 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:10.726 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:10.727 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:10.728 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:10.728 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:10.728 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:10.728 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:10.728 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:10.728 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:10.728 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.728 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:10.728 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.728 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.729 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:10.729 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.779 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:11.780 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:12.782 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:13.784 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:14.694 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:00:14.695 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.695 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:00:14.695 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.728 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:14.730 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:14.730 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:14.730 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:14.730 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:14.730 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:14.730 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:14.730 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:14.730 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.731 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:14.731 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.731 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.731 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:14.731 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.785 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:15.787 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:16.789 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:17.791 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:18.731 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:18.732 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:18.732 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:18.733 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:18.733 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:18.733 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:18.733 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:18.733 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:18.733 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.733 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:18.733 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.734 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.734 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:18.734 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.792 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:19.793 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:20.797 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:21.799 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:22.733 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:22.735 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:22.735 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:22.735 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:22.735 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:22.735 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:22.735 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:22.735 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:22.735 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.735 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:22.736 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.736 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.736 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:22.736 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.801 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:23.803 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:24.803 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:25.805 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:26.736 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:26.737 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:26.738 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:26.738 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:26.738 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:26.738 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:26.738 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:26.738 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:26.739 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.739 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:26.739 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.739 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.739 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:26.739 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.806 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:27.807 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:28.809 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:29.810 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:30.739 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:30.741 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:30.742 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:30.742 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:30.742 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:30.742 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:30.742 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:30.743 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:30.743 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.743 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:30.743 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.743 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.743 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:30.743 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.811 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:31.813 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:32.815 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:33.816 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:34.742 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:34.744 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:34.744 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:34.744 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:34.744 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:34.744 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:34.744 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:34.744 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:34.744 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.744 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:34.745 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.745 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.745 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:34.745 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.817 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:35.819 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:36.820 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:37.823 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:38.745 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:38.746 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:38.747 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:38.747 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:38.747 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:38.747 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:38.747 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:38.747 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:38.747 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.747 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:38.747 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.748 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.748 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:38.748 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.824 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:39.825 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:40.827 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:41.829 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:42.748 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:42.750 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:42.751 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:42.751 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:42.751 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:42.751 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:42.751 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:42.752 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:42.752 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.752 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:42.752 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.752 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.752 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:42.752 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.830 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:43.831 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:44.832 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:45.835 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:46.751 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:46.753 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:46.753 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:46.753 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:46.753 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:46.753 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:46.753 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:46.753 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:46.754 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.754 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:46.754 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.754 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.754 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:46.754 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.836 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:47.839 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:48.840 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:49.842 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:50.755 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:50.756 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:50.756 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:50.757 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:50.757 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:50.757 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:50.757 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:50.757 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:50.757 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.757 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:50.758 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.758 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.758 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:50.758 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.843 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:51.845 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:52.847 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:53.849 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:54.758 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:54.759 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:54.759 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:54.759 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:54.760 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:54.760 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:54.760 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:54.760 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:54.760 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.760 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:54.760 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.760 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.760 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:54.761 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.850 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:55.851 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:56.853 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:57.855 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:58.760 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:58.762 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:58.763 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:58.763 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:58.763 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:58.763 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:58.763 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:58.763 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:58.763 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.764 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:58.764 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.764 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.764 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:58.764 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.856 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:59.858 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:00.859 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:01.861 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:02.764 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:02.765 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:02.765 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:02.765 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:02.766 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:02.766 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:02.766 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:02.766 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:02.766 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.766 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:02.766 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.766 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.767 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:02.767 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.863 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:03.865 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:04.866 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:05.867 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:06.766 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:06.768 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:06.769 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:06.769 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:06.769 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:06.769 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:06.769 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:06.769 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:06.769 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.770 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:06.770 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.770 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.770 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:06.770 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.868 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:07.870 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:08.871 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:09.872 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:10.480 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-d6447694-7a63-4e16-8ab1-f2203b502079', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 02:01:10.602 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3995-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'], 'id_perms': {'created': '2025-09-01T02:01:10.595372', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:10.595372', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11952340681741785153, 'uuid_mslong': 6586627194885784908}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5b686689-4b18-454c-a5df-3e8619ab5841', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.192'}, 'oper': 'CREATE', 'request-id': 'req-681edb8a-0de7-491e-8459-bf06930acfe8', 'type': 'virtual_router', 'uuid': '5b686689-4b18-454c-a5df-3e8619ab5841'} 09/01/2025 02:01:10.769 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:10.770 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:10.770 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:10.770 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:10.770 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:10.770 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:10.771 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:10.771 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:10.771 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.771 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:10.771 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.771 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.771 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:10.771 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.874 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:10.940 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:01:10.925113', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:10.925113', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10735908003063543833, 'uuid_mslong': 17648289949859202616}}, 'parent_type': 'virtual-router', 'parent_uuid': '5b686689-4b18-454c-a5df-3e8619ab5841', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:f4:eb:52:34:6a']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'}]}, 'oper': 'CREATE', 'request-id': 'req-1c3ccdca-0f5f-4532-9c6f-20034282c8ff', 'type': 'virtual_machine_interface', 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819'} 09/01/2025 02:01:10.965 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-1c3ccdca-0f5f-4532-9c6f-20034282c8ff', 'type': 'virtual_machine_interface', 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819'} 09/01/2025 02:01:11.478 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-8b202558-6c4f-4e38-a46f-f0bfe3ec33be', 'type': 'access_control_list', 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'} 09/01/2025 02:01:11.875 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:12.876 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:13.878 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:14.695 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:01:14.695 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.695 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:01:14.695 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.771 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:14.772 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:14.772 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:14.772 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:14.772 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:14.772 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:14.772 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:14.772 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:14.772 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.773 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:14.773 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.773 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.773 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:14.773 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.879 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:15.880 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:16.882 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:17.884 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:18.773 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:18.774 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:18.775 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:18.775 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:18.775 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:18.775 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:18.775 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:18.775 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:18.775 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.775 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:18.775 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.776 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.776 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:18.776 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.885 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:19.886 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:20.887 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:21.888 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:22.776 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:22.777 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:22.777 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:22.777 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:22.778 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:22.778 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:22.778 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:22.778 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:22.778 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.778 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:22.778 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.778 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.778 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:22.778 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.890 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:23.891 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:24.892 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:25.893 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:26.778 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:26.779 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:26.779 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:26.779 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:26.779 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:26.779 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:26.779 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:26.779 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:26.780 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.780 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:26.780 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.780 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.780 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:26.780 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.894 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:27.896 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:28.897 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:29.898 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:30.780 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:30.781 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:30.781 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:30.781 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:30.781 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:30.781 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:30.781 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:30.782 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:30.782 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.782 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:30.782 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.782 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.782 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:30.782 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.899 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:31.901 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:32.902 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:33.905 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:34.783 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:34.784 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:34.784 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:34.784 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:34.784 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:34.784 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:34.785 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:34.785 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:34.785 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.785 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:34.785 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.785 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.785 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:34.785 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.907 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:35.908 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:36.910 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:37.911 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:38.785 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:38.786 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:38.786 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:38.786 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:38.786 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:38.787 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:38.787 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:38.787 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:38.787 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.787 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:38.787 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.787 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.787 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:38.787 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.913 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:39.914 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:40.915 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:41.917 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:42.787 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:42.789 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:42.789 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:42.789 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:42.790 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:42.790 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:42.790 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:42.790 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:42.790 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.790 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:42.790 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.790 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.791 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:42.791 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.919 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:43.920 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:44.922 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:45.923 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:46.791 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:46.792 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:46.792 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:46.792 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:46.793 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:46.793 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:46.793 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:46.793 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:46.793 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.793 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:46.793 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.793 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.793 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:46.793 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.925 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:47.926 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:48.927 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:49.928 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:50.793 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:50.794 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:50.794 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:50.794 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:50.794 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:50.794 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:50.795 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:50.795 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:50.795 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.795 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:50.795 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.795 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.795 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:50.795 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.930 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:51.930 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:52.932 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:53.934 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:54.795 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:54.796 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:54.796 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:54.796 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:54.796 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:54.796 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:54.796 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:54.797 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:54.797 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.797 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:54.797 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.797 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.797 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:54.797 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.935 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:55.936 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:56.938 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:57.940 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:58.796 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:58.798 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:58.798 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:58.798 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:58.798 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:58.798 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:58.798 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:58.799 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:58.799 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.799 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:58.799 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.799 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.799 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:58.799 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.941 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:59.942 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:00.944 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:01.944 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:02.799 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:02.800 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:02.800 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:02.800 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:02.801 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:02.801 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:02.801 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:02.801 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:02.801 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.801 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:02.801 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.801 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.801 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:02.802 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.946 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:02.994 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3995-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'], 'id_perms': {'created': '2025-09-01T02:02:02.985199', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:02.985199', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10627036196304171688, 'uuid_mslong': 16146534956283939804}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.18'}, 'oper': 'CREATE', 'request-id': 'req-7f54cc47-40c2-4259-85af-4adeb8e3ee0f', 'type': 'virtual_router', 'uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8'} 09/01/2025 02:02:03.334 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:02:03.316910', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:03.316910', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12779270811540185455, 'uuid_mslong': 492471125198063140}}, 'parent_type': 'virtual-router', 'parent_uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:06:d5:9b:e1:0f']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'}]}, 'oper': 'CREATE', 'request-id': 'req-e0b4c5ea-eae1-45cd-9751-832bb456eea9', 'type': 'virtual_machine_interface', 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f'} 09/01/2025 02:02:03.359 7f1bc3401540 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-e0b4c5ea-eae1-45cd-9751-832bb456eea9', 'type': 'virtual_machine_interface', 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f'} 09/01/2025 02:02:03.948 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:04.949 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:05.950 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:06.801 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:06.802 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:06.802 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:06.802 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:06.802 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:06.803 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:06.803 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:06.803 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:06.803 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.804 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:06.804 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.804 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.804 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:06.804 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.951 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:07.952 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:08.953 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:09.954 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:10.590 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:10.802 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:10.804 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:10.804 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:10.804 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:10.804 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:10.804 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:10.804 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:10.805 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:10.805 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.805 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:10.805 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.805 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.805 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:10.805 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.955 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:11.957 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:12.959 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:13.960 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:14.696 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:02:14.696 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.696 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:02:14.696 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.805 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:14.806 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:14.806 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:14.806 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:14.806 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:14.806 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:14.806 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:14.806 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:14.807 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.807 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:14.807 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.807 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.807 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:14.807 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.961 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:15.963 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:16.965 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:17.966 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:18.807 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:18.808 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:18.808 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:18.808 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:18.809 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:18.809 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:18.809 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:18.809 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:18.809 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.809 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:18.809 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.809 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.809 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:18.810 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.968 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:19.969 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:20.970 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:21.972 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:22.810 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:22.811 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:22.811 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:22.811 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:22.811 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:22.811 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:22.811 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:22.812 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:22.812 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.812 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:22.812 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.812 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.812 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:22.812 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.827 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:22.973 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:23.975 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:24.976 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:25.978 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:26.812 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:26.813 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:26.814 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:26.814 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:26.814 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:26.814 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:26.814 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:26.814 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:26.814 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.814 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:26.814 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.815 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.815 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:26.815 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.979 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:27.981 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:28.983 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:29.984 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:30.815 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:30.816 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:30.817 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:30.817 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:30.817 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:30.817 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:30.817 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:30.817 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:30.817 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.818 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:30.818 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.818 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.818 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:30.818 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.985 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:31.986 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:32.988 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:33.990 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:34.817 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:34.819 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:34.819 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:34.819 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:34.819 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:34.819 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:34.819 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:34.819 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:34.819 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.820 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:34.820 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.820 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.820 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:34.820 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.991 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:35.083 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:35.994 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:36.995 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:37.997 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:38.820 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:38.821 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:38.821 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:38.821 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:38.821 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:38.821 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:38.822 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:38.822 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:38.822 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.822 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:38.822 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.822 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.822 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:38.822 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.999 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:40.000 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:41.001 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:42.003 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:42.822 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:42.823 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:42.823 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:42.823 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:42.823 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:42.824 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:42.824 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:42.824 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:42.824 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.824 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:42.824 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.824 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.824 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:42.824 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:43.004 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:44.005 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:45.006 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:46.007 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:46.824 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:46.825 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:46.826 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:46.826 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:46.826 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:46.826 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:46.826 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:46.826 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:46.826 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.826 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:46.826 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.827 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.827 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:46.827 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:47.009 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:47.158 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:02:48.011 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:49.013 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:50.015 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:50.827 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:50.828 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:50.828 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:50.828 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:50.828 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:50.829 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:50.829 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:50.829 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:50.829 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.829 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:50.829 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.829 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.829 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:50.829 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:51.016 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:52.018 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:53.019 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:54.021 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:54.829 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:54.830 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:54.831 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:54.831 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:54.831 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:54.831 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:54.831 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:54.831 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:54.831 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.831 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:54.831 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.831 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.832 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:54.832 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:55.022 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:56.023 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:57.024 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:58.025 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:58.832 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:58.833 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:58.833 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:58.833 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:58.834 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:58.834 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:58.834 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:58.834 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:58.834 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.834 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:58.834 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.834 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.834 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:58.834 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:59.026 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:59.158 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:00.027 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:01.029 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:02.031 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:02.834 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:02.836 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:02.836 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:02.836 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:02.836 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:02.836 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:02.836 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:02.836 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:02.836 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.837 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:02.837 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.837 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.837 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:02.837 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:03.032 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:04.034 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:05.035 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:06.037 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:06.836 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:06.838 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:06.838 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:06.838 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:06.838 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:06.838 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:06.838 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:06.838 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:06.838 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.839 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:06.839 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.839 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.839 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:06.839 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:07.038 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:08.040 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:09.041 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:10.042 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:10.839 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:10.840 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:10.841 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:10.841 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:10.841 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:10.841 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:10.841 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:10.841 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:10.841 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.841 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:10.841 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.841 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.842 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:10.842 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:11.044 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:11.444 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:12.046 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:13.048 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:14.050 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:14.697 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:03:14.697 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.697 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:03:14.698 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.841 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:14.842 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:14.842 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:14.842 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:14.843 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:14.843 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:14.843 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:14.843 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:14.843 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.843 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:14.843 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.843 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.843 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:14.843 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:15.050 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:16.053 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:17.054 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:18.056 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:18.843 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:18.845 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:18.845 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:18.845 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:18.845 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:18.845 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:18.845 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:18.845 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:18.846 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.846 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:18.846 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.846 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.846 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:18.846 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:19.056 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:20.058 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:21.060 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:22.062 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:22.846 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:22.847 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:22.847 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:22.848 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:22.848 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:22.848 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:22.848 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:22.848 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:22.848 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.848 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:22.848 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.848 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.848 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:22.849 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:23.063 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:23.497 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:24.065 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:25.066 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:26.068 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:26.848 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:26.849 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:26.849 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:26.849 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:26.850 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:26.850 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:26.850 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:26.850 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:26.850 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.850 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:26.850 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.850 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.850 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:26.850 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:27.069 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:28.071 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:29.072 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:30.074 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:30.850 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:30.851 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:30.851 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:30.851 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:30.851 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:30.851 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:30.851 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:30.852 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:30.852 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.852 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:30.852 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.852 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.852 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:30.852 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:31.076 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:32.077 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:33.079 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:34.080 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:34.852 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:34.853 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:34.853 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:34.853 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:34.854 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:34.854 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:34.854 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:34.854 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:34.854 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.855 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:34.855 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.855 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.855 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:34.855 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:35.081 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:35.703 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:36.083 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:37.085 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:38.086 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:38.854 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:38.855 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:38.855 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:38.855 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:38.855 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:38.856 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:38.856 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:38.856 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:38.856 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.856 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:38.856 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.856 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.856 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:38.856 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:39.087 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:40.088 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:41.089 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:42.091 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:42.856 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:42.858 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:42.858 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:42.858 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:42.858 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:42.858 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:42.858 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:42.858 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:42.858 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.859 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:42.859 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.859 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.859 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:42.859 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:43.092 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:44.094 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:45.095 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:46.097 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:46.859 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:46.860 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:46.860 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:46.860 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:46.861 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:46.861 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:46.861 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:46.861 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:46.861 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.861 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:46.861 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.861 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.861 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:46.861 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:47.098 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:47.883 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:03:48.099 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:49.101 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:50.103 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:50.862 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:50.863 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:50.863 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:50.863 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:50.863 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:50.863 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:50.864 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:50.864 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:50.864 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.864 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:50.864 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.864 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.864 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:50.864 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:51.104 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:52.106 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:53.108 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:54.109 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:54.863 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:54.865 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:54.865 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:54.865 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:54.865 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:54.865 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:54.865 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:54.865 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:54.866 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.866 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:54.866 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.866 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.866 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:54.866 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:55.110 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:56.112 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:57.113 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:58.115 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:58.866 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:58.867 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:58.867 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:58.867 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:58.868 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:58.868 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:58.868 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:58.868 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:58.868 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.868 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:58.868 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.868 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.868 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:58.868 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:59.117 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:59.932 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:00.118 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:01.120 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:02.122 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:02.868 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:02.870 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:02.870 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:02.870 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:02.870 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:02.870 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:02.870 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:02.871 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:02.871 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.871 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:02.871 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.871 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.871 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:02.871 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:03.123 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:04.125 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:05.126 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:06.128 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:06.871 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:06.873 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:06.873 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:06.873 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:06.873 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:06.873 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:06.873 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:06.873 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:06.874 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.874 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:06.874 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.874 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.874 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:06.874 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:07.129 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:08.131 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:09.133 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:10.134 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:10.873 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:10.875 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:10.875 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:10.875 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:10.875 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:10.875 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:10.875 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:10.875 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:10.875 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.876 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:10.876 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.876 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.876 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:10.876 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:11.135 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:12.136 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:12.219 7f1bc33e6b80 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:13.138 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:14.139 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:14.694 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Starting the Kafka producer 09/01/2025 02:04:14.695 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name connections-closed 09/01/2025 02:04:14.696 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name connections-created 09/01/2025 02:04:14.696 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name select-time 09/01/2025 02:04:14.696 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name io-time 09/01/2025 02:04:14.696 7f1bc34eca40 [kafka.client] [DEBUG]: Attempting to check version with node bootstrap-2 09/01/2025 02:04:14.696 7f1bc34eca40 [kafka.client] [DEBUG]: Initiating connection to node bootstrap-2 at 10.0.0.131:9092 09/01/2025 02:04:14.696 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-sent-received 09/01/2025 02:04:14.696 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-sent 09/01/2025 02:04:14.696 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-received 09/01/2025 02:04:14.696 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name request-latency 09/01/2025 02:04:14.697 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name throttle-time 09/01/2025 02:04:14.697 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-2.bytes-sent 09/01/2025 02:04:14.697 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-2.bytes-received 09/01/2025 02:04:14.697 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-2.latency 09/01/2025 02:04:14.697 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-2.throttle 09/01/2025 02:04:14.697 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:04:14.698 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.698 7f1bc33e6b80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:04:14.698 7f1bc33e6b80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.698 7f1bc34eca40 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:04:14.698 7f1bc34eca40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:04:14.698 7f1bc34eca40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: connecting to 10.0.0.131:9092 [('10.0.0.131', 9092) IPv4] 09/01/2025 02:04:14.699 7f1bc34eca40 [kafka.client] [DEBUG]: Timeouts: user 199.925661, metadata inf, idle connection inf, request inf 09/01/2025 02:04:14.700 7f1bc34eca40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: established TCP connection 09/01/2025 02:04:14.700 7f1bc34eca40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: checking broker Api Versions 09/01/2025 02:04:14.700 7f1bc34eca40 [kafka.protocol.parser] [DEBUG]: Sending request ApiVersionsRequest_v4(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:14.700 7f1bc34eca40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 1 (timeout_ms 1600.0): ApiVersionsRequest_v4(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:14.701 7f1bc34eca40 [kafka.client] [DEBUG]: Timeouts: user 199.957132, metadata inf, idle connection inf, request 1599.508524 09/01/2025 02:04:14.701 7f1bc34eca40 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:04:14.702 7f1bc34eca40 [kafka.protocol.parser] [DEBUG]: Processing response ApiVersionsResponse_v4 09/01/2025 02:04:14.702 7f1bc34eca40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 1 (1.3997554779052734 ms): ApiVersionsResponse_v0(error_code=35, api_versions=[(api_key=18, min_version=0, max_version=3)]) 09/01/2025 02:04:14.702 7f1bc34eca40 [kafka.client] [DEBUG]: Timeouts: user 199.962139, metadata inf, idle connection 539992.000000, request inf 09/01/2025 02:04:14.702 7f1bc34eca40 [kafka.protocol.parser] [DEBUG]: Sending request ApiVersionsRequest_v3(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:14.703 7f1bc34eca40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 2 (timeout_ms 1600.0): ApiVersionsRequest_v3(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:14.703 7f1bc34eca40 [kafka.client] [DEBUG]: Timeouts: user 199.964523, metadata inf, idle connection 539992.000000, request 1599.671841 09/01/2025 02:04:14.704 7f1bc34eca40 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:04:14.704 7f1bc34eca40 [kafka.protocol.parser] [DEBUG]: Processing response ApiVersionsResponse_v3 09/01/2025 02:04:14.705 7f1bc34eca40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 2 (1.9135475158691406 ms): ApiVersionsResponse_v3(error_code=0, api_versions=[(api_key=0, min_version=0, max_version=8, _tagged_fields={}), (api_key=1, min_version=0, max_version=11, _tagged_fields={}), (api_key=2, min_version=0, max_version=5, _tagged_fields={}), (api_key=3, min_version=0, max_version=9, _tagged_fields={}), (api_key=4, min_version=0, max_version=4, _tagged_fields={}), (api_key=5, min_version=0, max_version=3, _tagged_fields={}), (api_key=6, min_version=0, max_version=6, _tagged_fields={}), (api_key=7, min_version=0, max_version=3, _tagged_fields={}), (api_key=8, min_version=0, max_version=8, _tagged_fields={}), (api_key=9, min_version=0, max_version=7, _tagged_fields={}), (api_key=10, min_version=0, max_version=3, _tagged_fields={}), (api_key=11, min_version=0, max_version=7, _tagged_fields={}), (api_key=12, min_version=0, max_version=4, _tagged_fields={}), (api_key=13, min_version=0, max_version=4, _tagged_fields={}), (api_key=14, min_version=0, max_version=5, _tagged_fields={}), (api_key=15, min_version=0, max_version=5, _tagged_fields={}), (api_key=16, min_version=0, max_version=4, _tagged_fields={}), (api_key=17, min_version=0, max_version=1, _tagged_fields={}), (api_key=18, min_version=0, max_version=3, _tagged_fields={}), (api_key=19, min_version=0, max_version=5, _tagged_fields={}), (api_key=20, min_version=0, max_version=4, _tagged_fields={}), (api_key=21, min_version=0, max_version=2, _tagged_fields={}), (api_key=22, min_version=0, max_version=3, _tagged_fields={}), (api_key=23, min_version=0, max_version=3, _tagged_fields={}), (api_key=24, min_version=0, max_version=1, _tagged_fields={}), (api_key=25, min_version=0, max_version=1, _tagged_fields={}), (api_key=26, min_version=0, max_version=1, _tagged_fields={}), (api_key=27, min_version=0, max_version=0, _tagged_fields={}), (api_key=28, min_version=0, max_version=3, _tagged_fields={}), (api_key=29, min_version=0, max_version=2, _tagged_fields={}), (api_key=30, min_version=0, max_version=2, _tagged_fields={}), (api_key=31, min_version=0, max_version=2, _tagged_fields={}), (api_key=32, min_version=0, max_version=3, _tagged_fields={}), (api_key=33, min_version=0, max_version=1, _tagged_fields={}), (api_key=34, min_version=0, max_version=1, _tagged_fields={}), (api_key=35, min_version=0, max_version=2, _tagged_fields={}), (api_key=36, min_version=0, max_version=2, _tagged_fields={}), (api_key=37, min_version=0, max_version=2, _tagged_fields={}), (api_key=38, min_version=0, max_version=2, _tagged_fields={}), (api_key=39, min_version=0, max_version=2, _tagged_fields={}), (api_key=40, min_version=0, max_version=2, _tagged_fields={}), (api_key=41, min_version=0, max_version=2, _tagged_fields={}), (api_key=42, min_version=0, max_version=2, _tagged_fields={}), (api_key=43, min_version=0, max_version=2, _tagged_fields={}), (api_key=44, min_version=0, max_version=1, _tagged_fields={}), (api_key=45, min_version=0, max_version=0, _tagged_fields={}), (api_key=46, min_version=0, max_version=0, _tagged_fields={}), (api_key=47, min_version=0, max_version=0, _tagged_fields={}), (api_key=48, min_version=0, max_version=0, _tagged_fields={}), (api_key=49, min_version=0, max_version=0, _tagged_fields={})], throttle_time_ms=0, _tagged_fields={}) 09/01/2025 02:04:14.706 7f1bc34eca40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: Broker version identified as 2.6 09/01/2025 02:04:14.706 7f1bc34eca40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: Connection complete. 09/01/2025 02:04:14.706 7f1bc34eca40 [kafka.client] [DEBUG]: Node bootstrap-2 connected 09/01/2025 02:04:14.707 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name batch-size 09/01/2025 02:04:14.707 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name compression-rate 09/01/2025 02:04:14.707 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name queue-time 09/01/2025 02:04:14.707 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name records-per-request 09/01/2025 02:04:14.708 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes 09/01/2025 02:04:14.708 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name record-retries 09/01/2025 02:04:14.708 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name errors 09/01/2025 02:04:14.708 7f1bc34eca40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name record-size-max 09/01/2025 02:04:14.708 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Starting Kafka producer I/O thread. 09/01/2025 02:04:14.709 7f1bc33e6b80 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) to node bootstrap-2 09/01/2025 02:04:14.709 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) 09/01/2025 02:04:14.709 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 3 (timeout_ms 30000): MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) 09/01/2025 02:04:14.709 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.388943, metadata 30000.000000, idle connection 539986.000000, request 29999.963999 09/01/2025 02:04:14.709 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Kafka producer started 09/01/2025 02:04:14.709 7f1bc34eca40 [contrail-alarm-gen] [INFO]: Initialized health-check KafkaProducer 09/01/2025 02:04:14.710 7f1bc34eca40 [contrail-alarm-gen] [INFO]: health-check KafkaProducer call returned 09/01/2025 02:04:14.710 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.989281, metadata 30000.000000, idle connection 539985.000000, request 29999.181271 09/01/2025 02:04:14.711 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 02:04:14.711 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:04:14.711 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 3 (2.2029876708984375 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[]) 09/01/2025 02:04:14.712 7f1bc33e6b80 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 3, topics: 0, groups: 0) 09/01/2025 02:04:14.712 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.989519, metadata 299999.757324, idle connection 539983.000000, request inf 09/01/2025 02:04:14.876 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:14.877 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:14.877 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:14.877 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:14.878 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:14.878 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:14.878 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:14.878 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:14.878 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.878 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:14.878 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.878 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.879 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:14.879 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:15.141 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:16.142 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:17.143 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:18.145 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:18.878 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:18.880 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:18.880 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:18.880 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:18.880 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:18.880 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:18.880 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:18.880 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:18.880 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.881 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:18.881 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.881 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.881 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:18.881 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:19.146 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:20.148 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:21.149 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:22.151 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:22.880 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:22.882 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:22.882 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:22.882 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:22.882 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:22.882 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:22.882 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:22.883 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:22.883 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.883 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:22.883 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.883 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.883 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:22.883 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:23.152 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:24.154 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:24.258 7f1bc33c9360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:25.155 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:26.156 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:26.883 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:26.884 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:26.884 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:26.885 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:26.885 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:26.885 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:26.885 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:26.885 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:26.885 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.885 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:26.885 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.885 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.885 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:26.886 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:27.157 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:28.159 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:29.159 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:30.161 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:30.885 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:30.887 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:30.887 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:30.887 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:30.887 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:30.887 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:30.887 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:30.888 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:30.888 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.888 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:30.888 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.888 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.888 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:30.888 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:31.162 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:32.164 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:33.166 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:34.168 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:34.887 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:34.889 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:34.889 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:34.889 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:34.889 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:34.889 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:34.889 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:34.889 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:34.890 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.890 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:34.890 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.890 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.890 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:34.890 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:35.169 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:36.218 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:36.582 7f1bc33c9360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:37.219 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:38.221 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:38.890 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:38.891 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:38.891 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:38.891 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:38.891 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:38.891 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:38.891 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:38.892 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:38.892 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.892 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:38.892 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.892 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.892 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:38.892 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:39.223 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:40.224 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:41.226 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:42.226 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:42.892 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:42.893 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:42.894 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:42.894 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:42.894 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:42.894 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:42.894 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:42.894 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:42.894 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.894 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:42.895 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.895 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.895 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:42.895 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:43.228 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:44.230 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:45.231 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:46.233 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:46.895 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:46.896 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:46.897 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:46.897 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:46.897 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:46.897 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:46.897 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:46.897 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:46.898 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.898 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:46.898 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.898 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.898 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:46.898 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:47.234 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:48.236 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:48.991 7f1bc33c9360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:49.238 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:50.239 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:50.898 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:50.899 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:50.899 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:50.899 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:50.899 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:50.899 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:50.899 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:50.900 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:50.900 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.900 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:50.900 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.900 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.900 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:50.900 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:51.240 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:52.242 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:53.244 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:54.246 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:54.900 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:54.902 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:54.902 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:54.902 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:54.902 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:54.902 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:54.902 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:54.902 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:54.903 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.903 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:54.903 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.903 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.903 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:54.903 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:55.247 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:56.248 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:57.250 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:58.252 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:58.903 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:58.905 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:58.905 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:58.905 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:58.905 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:58.905 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:58.905 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:58.905 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:58.905 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.906 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:58.906 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.906 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.906 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:58.906 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:59.253 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:00.255 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:01.059 7f1bc33c9360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:01.256 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:02.258 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:02.906 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:02.907 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:02.907 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:02.908 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:02.908 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:02.908 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:02.908 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:02.908 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:02.909 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.909 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:02.909 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.909 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.909 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:02.909 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:03.259 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:04.261 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:05.262 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:06.264 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:06.908 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:06.909 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:06.910 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:06.910 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:06.910 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:06.910 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:06.910 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:06.910 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:06.910 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.910 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:06.910 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.911 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.911 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:06.911 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:07.265 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:08.267 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:09.269 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:10.270 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:10.911 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:10.913 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:10.913 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:10.913 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:10.914 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:10.914 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:10.914 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:10.914 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:10.914 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.915 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:10.915 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.915 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.915 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:10.915 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:11.271 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:12.273 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:13.306 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:13.711 7f1bc33c9360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:14.308 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:14.699 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:05:14.699 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.699 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:05:14.699 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.913 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:14.915 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:14.915 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:14.915 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:14.915 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:14.915 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:14.915 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:14.915 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:14.915 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.916 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:14.916 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.916 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.916 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:14.916 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:15.309 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:16.311 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:17.313 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:18.314 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:18.916 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:18.917 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:18.918 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:18.918 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:18.918 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:18.918 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:18.918 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:18.918 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:18.918 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.918 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:18.918 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.918 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.919 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:18.919 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:19.316 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:20.318 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:21.320 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:22.322 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:22.919 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:22.920 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:22.920 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:22.920 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:22.920 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:22.920 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:22.921 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:22.921 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:22.921 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.921 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:22.921 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.921 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.921 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:22.921 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:23.323 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:24.325 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:25.326 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:25.743 7f1bc33c9360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:26.327 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:26.920 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:26.922 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:26.922 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:26.922 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:26.923 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:26.923 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:26.923 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:26.923 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:26.923 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.923 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:26.923 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.923 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.923 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:26.923 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:27.329 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:28.331 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:29.333 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:30.335 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:30.924 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:30.925 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:30.925 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:30.925 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:30.925 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:30.925 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:30.925 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:30.926 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:30.926 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.926 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:30.926 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.926 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.926 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:30.926 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:31.337 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:32.338 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:33.339 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:34.341 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:34.926 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:34.927 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:34.927 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:34.927 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:34.928 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:34.928 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:34.928 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:34.928 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:34.928 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.928 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:34.928 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.928 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.929 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:34.929 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:35.343 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:36.345 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:37.347 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:38.013 7f1bc33c9360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:38.349 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:38.928 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:38.929 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:38.929 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:38.929 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:38.930 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:38.930 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:38.930 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:38.930 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:38.930 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.930 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:38.930 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.930 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.930 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:38.930 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:39.350 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:40.352 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:41.353 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:42.355 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:42.930 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:42.932 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:42.932 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:42.932 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:42.932 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:42.932 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:42.932 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:42.933 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:42.933 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.933 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:42.933 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.933 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.933 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:42.933 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:43.357 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:44.359 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:45.361 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:46.363 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:46.933 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:46.935 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:46.935 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:46.935 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:46.935 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:46.935 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:46.936 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:46.936 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:46.936 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.936 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:46.936 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.936 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.936 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:46.936 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:47.364 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:48.366 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:49.367 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:50.250 7f1bc33c9360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:50.368 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:50.936 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:50.938 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:50.938 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:50.938 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:50.938 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:50.938 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:50.938 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:50.939 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:50.939 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.939 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:50.939 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.939 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.939 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:50.940 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:51.369 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:52.371 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:53.373 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:54.375 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:54.939 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:54.940 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:54.940 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:54.940 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:54.940 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:54.940 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:54.940 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:54.941 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:54.941 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.941 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:54.941 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.941 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.941 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:54.941 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:55.376 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:56.377 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:57.379 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:58.381 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:58.941 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:58.943 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:58.943 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:58.943 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:58.943 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:58.943 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:58.943 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:58.944 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:58.944 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.944 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:58.944 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.944 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.944 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:58.944 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:59.383 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:00.385 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:01.386 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:02.388 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:02.582 7f1bc33c9360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:02.944 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:02.946 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:02.946 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:02.946 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:02.946 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:02.946 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:02.947 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:02.947 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:02.947 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.947 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:02.948 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.948 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.948 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:02.948 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:03.390 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:04.391 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:05.393 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:06.395 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:06.946 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:06.948 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:06.948 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:06.948 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:06.948 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:06.948 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:06.948 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:06.948 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:06.948 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.949 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:06.949 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.949 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.949 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:06.949 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:07.396 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:08.398 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:09.400 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:10.402 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:10.949 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:10.950 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:10.950 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:10.950 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:10.951 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:10.951 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:10.951 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:10.951 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:10.951 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.951 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:10.951 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.951 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.951 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:10.951 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:11.403 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:12.405 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:13.407 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:14.408 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:14.699 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:06:14.700 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.700 7f1bc33c9360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:06:14.700 7f1bc33c9360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.710 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:14.711 7f1bc33e6b80 [kafka.client] [DEBUG]: Initializing connection to node 3 for metadata request 09/01/2025 02:06:14.711 7f1bc33e6b80 [kafka.client] [DEBUG]: Initiating connection to node 3 at 10.0.0.131:9092 09/01/2025 02:06:14.711 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-3.bytes-sent 09/01/2025 02:06:14.711 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-3.bytes-received 09/01/2025 02:06:14.711 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-3.latency 09/01/2025 02:06:14.712 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-3.throttle 09/01/2025 02:06:14.712 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:06:14.712 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:06:14.712 7f1bc33e6b80 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: connecting to 10.0.0.131:9092 [('10.0.0.131', 9092) IPv4] 09/01/2025 02:06:14.712 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999988.269329, metadata inf, idle connection 419982.000000, request inf 09/01/2025 02:06:14.713 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: established TCP connection 09/01/2025 02:06:14.713 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: checking broker Api Versions 09/01/2025 02:06:14.713 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Using pre-configured api_version (2, 6) for ApiVersions 09/01/2025 02:06:14.713 7f1bc33e6b80 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: Connection complete. 09/01/2025 02:06:14.713 7f1bc33e6b80 [kafka.client] [DEBUG]: Node 3 connected 09/01/2025 02:06:14.713 7f1bc33e6b80 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: Closing connection. 09/01/2025 02:06:14.713 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: reconnect backoff 0.05810463508383462 after 1 failures 09/01/2025 02:06:14.713 7f1bc33e6b80 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 3 09/01/2025 02:06:14.714 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.714 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 1 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.714 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.577770, metadata 30000.000000, idle connection 419981.000000, request 29999.976158 09/01/2025 02:06:14.714 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992380, metadata 30000.000000, idle connection 419981.000000, request 29999.679327 09/01/2025 02:06:14.730 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:06:14.730 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:14.730 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 1 (16.45970344543457 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:14.731 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.989281, metadata 99.956299, idle connection 419964.000000, request inf 09/01/2025 02:06:14.731 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.020566463470458984 secs. 09/01/2025 02:06:14.731 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:14.731 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992619, metadata 99.348389, idle connection 419964.000000, request inf 09/01/2025 02:06:14.832 7f1bc33e6b80 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 3 09/01/2025 02:06:14.832 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.832 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 2 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.833 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.328861, metadata 30000.000000, idle connection 419862.000000, request 29999.966860 09/01/2025 02:06:14.833 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990711, metadata 30000.000000, idle connection 419862.000000, request 29999.561071 09/01/2025 02:06:14.841 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:06:14.841 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:14.841 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 2 (8.511781692504883 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:14.841 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990950, metadata 99.962158, idle connection 419853.000000, request inf 09/01/2025 02:06:14.842 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.1314067840576172 secs. 09/01/2025 02:06:14.842 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:14.842 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992619, metadata 99.437256, idle connection 419853.000000, request inf 09/01/2025 02:06:14.943 7f1bc33e6b80 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 3 09/01/2025 02:06:14.943 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.943 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 3 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.943 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.372730, metadata 30000.000000, idle connection 419752.000000, request 29999.969244 09/01/2025 02:06:14.944 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991903, metadata 30000.000000, idle connection 419751.000000, request 29999.584436 09/01/2025 02:06:14.950 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 02:06:14.950 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:14.951 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 3 (7.557868957519531 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=5, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=10, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=20, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=25, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=26, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=11, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=4, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=16, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=17, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=23, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=28, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=13, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=22, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=8, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=2, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=29, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=19, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=14, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=1, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=7, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[])])]) 09/01/2025 02:06:14.952 7f1bc33e6b80 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 3, topics: 1, groups: 0) 09/01/2025 02:06:14.952 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.987135, metadata 299999.823975, idle connection 419742.000000, request inf 09/01/2025 02:06:14.952 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.2423112392425537 secs. 09/01/2025 02:06:14.953 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) 09/01/2025 02:06:14.953 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) is either full or getting a new batch 09/01/2025 02:06:14.953 7f1bc34eca40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:06:14.953 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:06:14.953 7f1bc34eca40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) 09/01/2025 02:06:14.954 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Node 1 not ready; delaying produce of accumulated batch (0.000000 ms) 09/01/2025 02:06:14.954 7f1bc33e6b80 [kafka.client] [DEBUG]: Initiating connection to node 1 at 10.0.0.249:9092 09/01/2025 02:06:14.954 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.bytes-sent 09/01/2025 02:06:14.954 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.bytes-received 09/01/2025 02:06:14.954 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.latency 09/01/2025 02:06:14.955 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.throttle 09/01/2025 02:06:14.955 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:14.955 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:06:14.956 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:06:14.956 7f1bc33e6b80 [kafka.conn] [INFO]: [IPv4 ('10.0.0.249', 9092)]>: connecting to 10.0.0.249:9092 [('10.0.0.249', 9092) IPv4] 09/01/2025 02:06:14.956 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 299995.684814, idle connection 419738.000000, request inf 09/01/2025 02:06:14.957 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Node 1 not ready; delaying produce of accumulated batch (47.920990 ms) 09/01/2025 02:06:14.957 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 47.909975, metadata 299995.319336, idle connection 419738.000000, request inf 09/01/2025 02:06:14.957 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:14.957 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:14.957 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:14.958 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:14.958 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:14.958 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:14.958 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:14.958 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.959 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:14.959 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.959 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.959 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:14.959 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.959 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: established TCP connection 09/01/2025 02:06:14.959 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: checking broker Api Versions 09/01/2025 02:06:14.959 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: Using pre-configured api_version (2, 6) for ApiVersions 09/01/2025 02:06:14.959 7f1bc33e6b80 [kafka.conn] [INFO]: [IPv4 ('10.0.0.249', 9092)]>: Connection complete. 09/01/2025 02:06:14.960 7f1bc33e6b80 [kafka.client] [DEBUG]: Node 1 connected 09/01/2025 02:06:14.960 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.records-per-batch 09/01/2025 02:06:14.960 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.bytes 09/01/2025 02:06:14.960 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.compression-rate 09/01/2025 02:06:14.960 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.record-retries 09/01/2025 02:06:14.960 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.record-errors 09/01/2025 02:06:14.960 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:06:14.960 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xc8\xab\x00^\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06e\xa9\x00\x00\x01\x99\x03\x06e\xa9\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:06:14.961 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xc8\xab\x00^\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06e\xa9\x00\x00\x01\x99\x03\x06e\xa9\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:06:14.961 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xc8\xab\x00^\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06e\xa9\x00\x00\x01\x99\x03\x06e\xa9\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:06:14.962 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: Request 1 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xc8\xab\x00^\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06e\xa9\x00\x00\x01\x99\x03\x06e\xa9\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:06:14.962 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 299990.433838, idle connection 419733.000000, request 29999.957085 09/01/2025 02:06:14.962 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992619, metadata 299990.073975, idle connection 419733.000000, request 29999.603271 09/01/2025 02:06:15.003 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:15.027 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:06:15.027 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:06:15.028 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: Response 1 (65.78612327575684 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:06:15.028 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:06:15.028 7f1bc33e6b80 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) with base offset 0 log start offset 0. 09/01/2025 02:06:15.028 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.988565, metadata 299924.073242, idle connection 419667.000000, request inf 09/01/2025 02:06:15.028 7f1bc34eca40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:06:15.410 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:16.412 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:17.414 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:18.415 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:18.959 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:18.960 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:18.961 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:18.961 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:18.961 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:18.961 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:18.961 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:18.961 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:18.962 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.962 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:18.962 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.962 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.962 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:18.962 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:19.417 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:20.418 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:21.420 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:22.422 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:22.961 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:22.963 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:22.963 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:22.963 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:22.963 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:22.963 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:22.963 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:22.963 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:22.963 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.964 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:22.964 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.964 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.964 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:22.964 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:23.423 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:24.425 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:25.426 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:26.428 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:26.964 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:26.965 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:26.965 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:26.965 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:26.965 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:26.965 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:26.965 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:26.965 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:26.965 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.966 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:26.966 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.966 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.966 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:26.966 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:27.266 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:27.430 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:28.458 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:29.459 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:30.460 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:30.966 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:30.967 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:30.967 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:30.967 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:30.967 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:30.967 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:30.967 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:30.968 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:30.968 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.968 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:30.968 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.968 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.968 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:30.968 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:31.462 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:32.463 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:33.464 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:34.466 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:34.967 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:34.968 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:34.968 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:34.969 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:34.969 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:34.969 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:34.969 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:34.969 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:34.969 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.969 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:34.969 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.969 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.969 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:34.970 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:35.467 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:36.468 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:37.500 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:38.502 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:38.970 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:38.971 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:38.971 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:38.971 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:38.971 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:38.971 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:38.971 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:38.971 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:38.971 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.972 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:38.972 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.972 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.972 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:38.972 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:39.430 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:39.502 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:40.503 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:41.529 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:42.529 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:42.972 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:42.973 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:42.973 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:42.973 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.973 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:42.973 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.973 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:42.973 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:42.974 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.974 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:42.974 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.974 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.974 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:42.974 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:43.531 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:44.532 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:45.533 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:46.534 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:46.974 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:46.976 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:46.976 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:46.976 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.976 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:46.976 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.976 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:46.977 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:46.977 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.977 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:46.977 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.977 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.977 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:46.977 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:47.535 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:48.536 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:49.538 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:50.539 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:50.977 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:50.979 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:50.979 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:50.979 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.979 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:50.979 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.979 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:50.980 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:50.980 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.980 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:50.980 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.980 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.980 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:50.980 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:51.430 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:51.541 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:52.542 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:53.544 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:54.544 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:54.980 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:54.981 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:54.981 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:54.981 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.981 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:54.982 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.982 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:54.982 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:54.982 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.982 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:54.982 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.982 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.982 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:54.982 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:55.546 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:56.548 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:57.549 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:58.550 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:58.982 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:58.983 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:58.983 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:58.983 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.983 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:58.983 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.983 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:58.984 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:58.984 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.984 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:58.984 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.984 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.984 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:58.984 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:59.551 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:00.552 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:01.553 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:02.555 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:02.984 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:02.985 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:02.985 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:02.985 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.985 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:02.986 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.986 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:02.986 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:02.986 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.986 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:02.986 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.986 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.986 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:02.987 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:03.496 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:03.556 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:04.557 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:05.559 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:06.560 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:06.986 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:06.988 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:06.988 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:06.988 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.988 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:06.989 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.989 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:06.989 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:06.989 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.989 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:06.989 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.990 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.990 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:06.990 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:07.578 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:08.580 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:09.581 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:10.582 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:10.988 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:10.990 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:10.990 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:10.990 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.990 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:10.990 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.990 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:10.991 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:10.991 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.991 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:10.991 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.991 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.991 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:10.991 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:11.584 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:12.585 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:13.587 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:14.588 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:14.700 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:07:14.762 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.763 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:07:14.763 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.990 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:14.991 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:14.992 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:14.992 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.992 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:14.992 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.992 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:14.992 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:14.992 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.992 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:14.992 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.992 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.993 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:14.993 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:15.589 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:16.086 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:16.590 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:17.591 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:18.592 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:18.992 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:18.994 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:18.994 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:18.994 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.995 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:18.995 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.995 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:18.995 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:18.995 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.995 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:18.995 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.995 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.995 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:18.995 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:19.594 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:20.596 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:21.597 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:22.599 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:22.995 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:22.996 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:22.996 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:22.996 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.997 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:22.997 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.997 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:22.997 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:22.997 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.997 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:22.997 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.997 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.997 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:22.998 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:23.600 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:24.601 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:25.604 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:26.605 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:26.998 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:26.999 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:26.999 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:26.999 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:27.000 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:27.000 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:27.000 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:27.000 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:27.000 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:27.000 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:27.000 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:27.000 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:27.001 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:27.001 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:27.607 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:28.222 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:28.608 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:29.610 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:30.611 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:31.000 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:31.002 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:31.002 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:31.002 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:31.003 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:31.003 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:31.003 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:31.003 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:31.003 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.003 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:31.004 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.004 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.004 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:31.004 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.613 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:32.615 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:33.616 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:34.618 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:35.003 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:35.005 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:35.005 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:35.005 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:35.005 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:35.005 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:35.005 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:35.006 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:35.006 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.006 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:35.006 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.006 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.006 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:35.006 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.620 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:36.621 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:37.623 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:38.625 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:39.006 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:39.007 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:39.007 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:39.007 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:39.007 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:39.008 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:39.008 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:39.008 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:39.008 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.008 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:39.008 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.009 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.009 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:39.009 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.626 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:40.241 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:40.627 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:41.629 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:42.630 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:43.008 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:43.010 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:43.010 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:43.010 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:43.010 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:43.010 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:43.010 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:43.011 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:43.011 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.011 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:43.011 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.011 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.011 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:43.011 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.632 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:44.634 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:45.635 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:46.636 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:47.011 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:47.012 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:47.013 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:47.013 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:47.013 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:47.013 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:47.013 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:47.013 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:47.013 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.014 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:47.014 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.014 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.014 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:47.014 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.638 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:48.639 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:49.641 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:50.643 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:51.014 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:51.016 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:51.016 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:51.016 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:51.016 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:51.016 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:51.016 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:51.016 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:51.016 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.017 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:51.017 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.017 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.017 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:51.017 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.644 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:52.577 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:52.645 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:53.647 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:54.648 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:55.016 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:55.018 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:55.018 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:55.018 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:55.018 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:55.018 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:55.018 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:55.019 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:55.019 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.019 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:55.019 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.019 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.019 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:55.019 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.649 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:56.651 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:57.652 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:58.654 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:59.019 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:59.021 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:59.021 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:59.021 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:59.021 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:59.021 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:59.021 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:59.022 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:59.022 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.022 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:59.022 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.022 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.022 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:59.022 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.655 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:00.656 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:01.658 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:02.660 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:03.022 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:03.024 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:03.024 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:03.024 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:03.024 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:03.024 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:03.024 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:03.024 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:03.024 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.025 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:03.025 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.025 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.025 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:03.025 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.661 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:04.662 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:04.704 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:05.664 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:06.665 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:07.024 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:07.055 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:07.055 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:07.055 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:07.055 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:07.055 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:07.055 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:07.056 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:07.056 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.056 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:07.056 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.056 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.056 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:07.056 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.667 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:08.669 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:09.670 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:10.671 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:11.056 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:11.057 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:11.057 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:11.057 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:11.057 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:11.057 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:11.058 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:11.058 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:11.058 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.058 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:11.058 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.058 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.058 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:11.058 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.672 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:12.673 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:13.675 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:14.676 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:14.700 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:08:14.701 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.701 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:08:14.701 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.029 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=21) 09/01/2025 02:08:15.030 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=21) is either full or getting a new batch 09/01/2025 02:08:15.030 7f1bc34eca40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:08:15.030 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:08:15.030 7f1bc34eca40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=21) 09/01/2025 02:08:15.030 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:08:15.031 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=21, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x827o\xca\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08:\xb6\x00\x00\x01\x99\x03\x08:\xb6\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:08:15.031 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=21, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x827o\xca\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08:\xb6\x00\x00\x01\x99\x03\x08:\xb6\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:08:15.031 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=21, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x827o\xca\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08:\xb6\x00\x00\x01\x99\x03\x08:\xb6\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:08:15.031 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: Request 2 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=21, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x827o\xca\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08:\xb6\x00\x00\x01\x99\x03\x08:\xb6\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:08:15.031 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 179921.065430, idle connection 299664.000000, request 29999.961376 09/01/2025 02:08:15.032 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992380, metadata 179920.637207, idle connection 299663.000000, request 29999.538183 09/01/2025 02:08:15.035 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:08:15.035 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:08:15.035 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: Response 2 (4.305124282836914 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=21, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:08:15.036 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=21, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:08:15.036 7f1bc33e6b80 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=21) with base offset 0 log start offset 0. 09/01/2025 02:08:15.036 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992380, metadata 179916.355957, idle connection 299659.000000, request inf 09/01/2025 02:08:15.036 7f1bc34eca40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:08:15.057 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:15.058 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:15.058 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:15.058 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:15.058 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:15.059 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:15.059 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:15.059 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:15.059 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.059 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:15.059 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.059 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.059 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:15.059 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.677 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:16.679 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:17.161 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:17.680 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:18.681 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:19.060 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:19.061 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:19.061 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:19.061 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:19.062 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:19.062 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:19.062 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:19.062 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:19.062 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.062 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:19.063 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.063 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.063 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:19.063 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.683 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:20.684 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:21.685 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:22.687 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:23.062 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:23.064 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:23.064 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:23.064 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:23.064 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:23.064 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:23.064 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:23.065 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:23.065 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.065 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:23.065 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.065 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.065 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:23.065 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.689 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:24.690 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:25.691 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:26.693 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:27.064 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:27.065 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:27.065 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:27.066 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:27.066 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:27.066 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:27.066 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:27.066 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:27.066 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.066 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:27.066 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.066 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.067 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:27.067 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.695 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:28.696 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:29.153 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:29.698 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:30.699 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:31.066 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:31.067 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:31.067 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:31.067 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:31.067 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:31.067 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:31.067 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:31.068 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:31.068 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.068 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:31.068 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.068 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.068 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:31.068 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.701 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:32.702 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:33.704 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:34.706 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:35.068 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:35.069 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:35.069 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:35.069 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:35.069 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:35.070 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:35.070 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:35.070 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:35.070 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.070 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:35.070 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.070 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.070 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:35.070 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.707 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:36.709 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:37.710 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:38.711 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:39.070 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:39.071 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:39.071 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:39.071 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:39.071 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:39.071 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:39.071 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:39.072 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:39.072 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.072 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:39.072 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.072 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.072 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:39.072 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.712 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:40.714 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:41.321 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:41.715 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:42.716 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:43.072 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:43.073 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:43.074 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:43.074 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:43.074 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:43.074 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:43.074 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:43.074 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:43.074 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.075 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:43.075 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.075 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.075 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:43.075 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.719 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:44.720 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:45.721 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:46.723 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:47.074 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:47.075 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:47.075 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:47.075 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:47.075 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:47.075 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:47.075 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:47.076 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:47.076 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.076 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:47.076 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.076 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.076 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:47.076 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.724 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:48.725 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:49.726 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:50.728 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:51.076 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:51.078 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:51.078 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:51.079 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:51.079 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:51.079 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:51.079 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:51.080 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:51.080 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.080 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:51.080 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.080 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.080 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:51.081 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.729 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:52.731 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:53.585 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:53.732 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:54.733 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:55.079 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:55.081 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:55.081 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:55.081 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:55.081 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:55.081 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:55.081 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:55.081 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:55.081 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.082 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:55.082 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.082 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.082 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:55.082 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.735 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:56.736 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:57.736 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:58.738 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:59.082 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:59.083 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:59.083 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:59.083 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:59.083 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:59.083 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:59.083 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:59.084 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:59.084 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.084 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:59.084 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.084 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.084 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:59.084 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.739 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:00.740 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:01.741 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:02.743 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:03.084 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:03.085 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:03.085 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:03.085 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:03.085 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:03.085 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:03.086 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:03.086 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:03.086 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.086 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:03.086 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.086 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.086 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:03.086 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.744 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:04.745 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:05.615 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:05.747 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:06.748 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:07.086 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:07.087 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:07.087 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:07.087 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:07.087 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:07.087 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:07.087 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:07.088 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:07.088 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.088 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:07.088 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.088 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.088 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:07.088 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.750 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:08.752 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:09.753 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:10.755 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:11.088 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:11.089 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:11.089 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:11.089 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:11.090 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:11.090 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:11.090 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:11.090 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:11.090 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.090 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:11.090 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.090 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.090 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:11.090 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.756 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:12.757 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:13.759 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:14.702 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:09:14.702 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.702 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:09:14.702 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.760 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:15.090 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:15.091 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:15.091 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:15.091 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:15.092 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:15.092 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:15.092 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:15.092 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:15.092 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.092 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:15.092 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.092 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.092 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:15.092 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.762 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:16.763 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:17.764 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:17.939 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:18.765 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:19.092 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:19.093 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:19.093 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:19.093 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:19.093 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:19.093 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:19.093 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:19.093 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:19.094 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.094 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:19.094 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.094 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.094 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:19.094 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.767 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:20.768 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:21.769 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:22.771 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:23.094 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:23.095 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:23.095 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:23.095 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:23.095 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:23.096 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:23.096 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:23.096 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:23.096 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.096 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:23.096 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.096 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.096 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:23.096 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.772 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:24.773 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:25.775 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:26.776 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:27.096 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:27.098 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:27.098 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:27.098 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:27.098 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:27.098 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:27.098 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:27.099 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:27.099 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.099 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:27.099 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.099 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.099 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:27.099 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.777 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:28.779 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:29.780 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:29.975 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:30.781 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:31.098 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:31.100 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:31.100 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:31.100 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:31.100 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:31.100 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:31.100 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:31.100 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:31.101 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.101 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:31.101 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.101 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.101 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:31.101 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.783 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:32.785 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:33.786 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:34.788 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:35.101 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:35.102 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:35.102 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:35.102 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:35.102 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:35.102 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:35.102 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:35.103 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:35.103 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.103 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:35.103 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.103 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.103 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:35.103 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.789 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:36.790 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:37.791 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:38.793 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:39.103 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:39.104 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:39.104 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:39.104 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:39.105 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:39.105 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:39.105 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:39.105 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:39.105 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.105 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:39.105 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.105 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.105 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:39.105 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.794 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:40.796 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:41.797 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:42.084 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:42.799 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:43.104 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:43.142 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:43.142 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:43.143 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:43.143 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:43.143 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:43.143 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:43.143 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:43.143 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.143 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:43.143 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.143 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.144 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:43.144 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.800 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:44.801 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:45.802 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:46.804 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:47.143 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:47.144 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:47.145 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:47.145 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:47.145 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:47.145 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:47.145 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:47.145 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:47.145 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.145 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:47.146 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.146 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.146 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:47.146 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.805 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:48.806 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:49.808 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:50.810 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:51.146 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:51.148 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:51.148 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:51.148 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:51.148 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:51.148 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:51.148 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:51.149 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:51.149 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.149 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:51.149 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.149 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.149 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:51.150 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.811 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:52.812 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:53.814 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:54.322 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:54.815 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:55.149 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:55.150 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:55.150 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:55.150 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:55.150 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:55.150 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:55.150 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:55.151 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:55.151 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.151 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:55.151 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.151 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.151 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:55.151 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.817 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:56.818 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:57.820 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:58.821 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:59.151 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:59.152 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:59.152 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:59.152 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:59.153 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:59.153 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:59.153 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:59.153 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:59.153 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.153 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:59.153 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.153 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.153 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:59.154 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.822 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:00.823 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:01.825 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:02.827 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:03.153 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:03.154 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:03.154 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:03.155 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:03.155 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:03.155 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:03.155 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:03.155 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:03.155 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.155 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:03.155 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.156 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.156 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:03.156 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.828 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:04.830 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:05.832 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:06.577 7f1bc2d12cc0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:06.834 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:07.155 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:07.157 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:07.157 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:07.157 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:07.157 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:07.157 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:07.157 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:07.158 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:07.158 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.158 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:07.158 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.158 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.158 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:07.158 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.835 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:08.836 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:09.838 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:10.840 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:11.158 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:11.160 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:11.160 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:11.160 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:11.160 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:11.160 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:11.160 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:11.161 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:11.161 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.161 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:11.161 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.161 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.161 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:11.161 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.841 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:12.842 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:13.843 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:14.703 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:10:14.703 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.704 7f1bc2d12cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:10:14.704 7f1bc2d12cc0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.845 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:15.037 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=27) 09/01/2025 02:10:15.037 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=27) is either full or getting a new batch 09/01/2025 02:10:15.037 7f1bc34eca40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:10:15.037 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:10:15.037 7f1bc34eca40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=27) 09/01/2025 02:10:15.038 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:10:15.038 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=27, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa9\x85{\xd4\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n\x0f}\x00\x00\x01\x99\x03\n\x0f}\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:10:15.038 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=27, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa9\x85{\xd4\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n\x0f}\x00\x00\x01\x99\x03\n\x0f}\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:10:15.038 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=27, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa9\x85{\xd4\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n\x0f}\x00\x00\x01\x99\x03\n\x0f}\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:10:15.038 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: Request 3 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=27, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa9\x85{\xd4\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n\x0f}\x00\x00\x01\x99\x03\n\x0f}\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:10:15.039 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 59913.663574, idle connection 179656.000000, request 29999.956608 09/01/2025 02:10:15.039 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992857, metadata 59913.218018, idle connection 179656.000000, request 29999.518633 09/01/2025 02:10:15.042 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 02:10:15.042 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:10:15.042 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: Response 3 (3.779888153076172 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=27, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:10:15.042 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=27, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:10:15.043 7f1bc33e6b80 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=27) with base offset 0 log start offset 0. 09/01/2025 02:10:15.043 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992142, metadata 59909.438965, idle connection 179652.000000, request inf 09/01/2025 02:10:15.043 7f1bc34eca40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:10:15.160 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:15.161 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:15.161 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:15.161 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:15.162 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:15.162 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:15.162 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:15.162 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:15.162 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.162 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:15.162 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.162 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.162 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:15.163 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.847 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:16.848 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:17.850 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:18.850 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:19.094 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:19.162 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:19.163 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:19.163 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:19.163 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:19.164 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:19.164 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:19.164 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:19.164 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:19.164 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.164 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:19.164 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.164 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.164 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:19.164 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.852 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:20.854 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:21.855 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:22.856 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:23.165 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:23.167 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:23.167 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:23.167 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:23.167 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:23.167 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:23.167 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:23.168 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:23.168 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.168 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:23.168 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.168 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.168 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:23.168 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.858 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:24.859 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:25.861 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:26.863 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:27.168 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:27.169 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:27.169 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:27.169 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:27.169 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:27.169 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:27.169 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:27.170 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:27.170 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.170 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:27.170 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.170 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.170 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:27.170 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.864 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:28.866 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:29.868 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:30.883 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:31.170 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:31.172 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:31.172 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:31.172 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:31.172 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:31.172 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:31.172 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:31.173 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:31.173 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.173 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:31.173 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.173 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.173 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:31.173 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.210 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:31.884 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:32.886 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:33.887 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:34.888 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:35.172 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:35.174 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:35.174 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:35.174 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:35.174 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:35.174 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:35.174 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:35.174 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:35.174 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.175 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:35.175 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.175 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.175 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:35.175 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.890 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:36.891 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:37.892 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:38.894 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:39.175 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:39.176 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:39.176 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:39.176 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:39.176 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:39.176 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:39.176 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:39.177 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:39.177 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.177 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:39.177 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.177 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.177 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:39.177 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.895 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:40.896 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:41.898 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:42.899 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:43.177 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:43.179 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:43.179 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:43.179 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:43.179 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:43.179 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:43.179 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:43.180 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:43.180 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.180 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:43.180 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.180 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.180 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:43.180 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.651 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:43.900 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:44.902 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:45.903 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:46.904 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:47.180 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:47.181 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:47.182 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:47.182 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:47.182 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:47.182 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:47.182 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:47.182 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:47.182 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.183 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:47.183 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.183 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.183 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:47.183 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.907 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:48.908 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:49.910 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:50.911 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:51.183 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:51.184 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:51.184 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:51.184 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:51.184 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:51.184 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:51.185 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:51.185 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:51.185 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.185 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:51.185 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.185 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.185 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:51.185 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.913 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:52.914 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:53.915 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:54.916 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:55.185 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:55.187 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:55.187 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:55.187 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:55.187 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:55.187 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:55.187 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:55.187 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:55.187 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.188 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:55.188 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.188 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.188 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:55.188 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.734 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:55.918 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:56.927 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:57.928 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:58.930 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:59.188 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:59.189 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:59.189 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:59.189 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:59.189 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:59.189 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:59.189 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:59.190 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:59.190 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.190 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:59.190 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.190 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.190 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:59.190 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.931 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:00.932 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:01.934 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:02.936 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:03.190 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:03.191 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:03.192 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:03.192 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:03.192 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:03.192 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:03.192 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:03.192 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:03.192 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.192 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:03.193 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.193 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.193 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:03.193 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.937 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:04.938 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:05.940 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:06.941 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:07.192 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:07.194 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:07.194 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:07.194 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:07.194 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:07.194 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:07.194 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:07.195 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:07.195 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.195 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:07.195 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.195 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.195 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:07.195 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.942 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:08.079 7f1bc2d12d60 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:08.944 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:09.945 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:10.946 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:11.194 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:11.196 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:11.196 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:11.196 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:11.196 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:11.196 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:11.197 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:11.197 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:11.197 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.197 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:11.197 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.197 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.197 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:11.197 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.947 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:12.948 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:13.950 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:14.704 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:11:14.705 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.705 7f1bc2d12d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:11:14.705 7f1bc2d12d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.951 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:14.953 7f1bc33e6b80 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:11:14.953 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:11:14.953 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: Request 4 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:11:14.953 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.465237, metadata 30000.000000, idle connection 119741.000000, request 29999.968290 09/01/2025 02:11:14.954 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990950, metadata 30000.000000, idle connection 119741.000000, request 29999.494791 09/01/2025 02:11:14.958 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Received correlation id: 4 09/01/2025 02:11:14.959 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:11:14.959 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.249', 9092)]>: Response 4 (5.868434906005859 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=5, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=10, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=20, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=25, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=26, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=11, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=4, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=16, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=17, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=23, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=28, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=13, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=22, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=8, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=2, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=29, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=19, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=14, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=1, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=7, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[])])]) 09/01/2025 02:11:14.960 7f1bc33e6b80 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 3, topics: 1, groups: 0) 09/01/2025 02:11:14.960 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991903, metadata 299999.871826, idle connection 119734.000000, request inf 09/01/2025 02:11:15.197 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:15.198 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:15.198 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:15.198 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:15.198 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:15.198 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:15.198 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:15.199 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:15.199 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.199 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:15.199 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.199 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.199 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:15.199 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.952 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:16.953 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:17.955 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:18.956 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:19.199 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:19.201 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:19.201 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:19.201 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:19.202 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:19.202 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:19.202 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:19.202 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:19.202 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.202 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:19.202 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.202 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.202 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:19.203 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.958 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:20.527 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:20.959 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:21.960 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:22.961 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:23.202 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:23.204 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:23.204 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:23.204 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:23.204 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:23.204 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:23.204 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:23.204 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:23.204 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.204 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:23.205 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.205 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.205 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:23.205 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.964 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:24.965 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:25.965 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:26.967 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:27.204 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:27.205 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:27.206 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:27.206 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:27.206 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:27.206 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:27.206 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:27.206 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:27.206 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.207 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:27.207 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.207 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.207 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:27.207 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.968 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:28.970 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:29.971 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:30.972 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:31.206 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:31.207 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:31.207 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:31.207 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:31.208 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:31.208 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:31.208 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:31.208 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:31.208 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.208 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:31.208 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.208 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.208 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:31.209 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.973 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:32.784 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:32.975 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:33.977 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:34.977 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:35.208 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:35.209 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:35.210 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:35.210 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:35.210 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:35.210 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:35.210 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:35.210 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:35.210 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.210 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:35.210 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.211 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.211 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:35.211 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.979 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:36.981 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:37.982 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:38.983 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:39.210 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:39.211 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:39.211 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:39.211 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:39.211 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:39.211 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:39.211 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:39.212 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:39.212 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.212 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:39.212 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.212 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.212 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:39.212 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.984 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:40.986 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:41.987 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:42.988 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:43.212 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:43.213 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:43.214 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:43.214 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:43.214 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:43.214 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:43.214 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:43.214 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:43.214 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.214 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:43.214 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.215 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.215 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:43.215 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.991 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:44.990 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:44.991 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:45.992 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:46.993 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:47.214 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:47.216 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:47.216 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:47.216 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:47.216 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:47.216 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:47.216 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:47.216 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:47.217 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.217 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:47.217 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.217 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.217 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:47.217 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.995 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:48.996 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:49.997 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:50.998 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:51.216 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:51.217 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:51.217 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:51.217 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:51.218 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:51.218 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:51.218 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:51.218 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:51.218 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.218 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:51.218 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.218 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.218 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:51.219 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.999 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:53.001 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:54.003 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:55.004 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:55.218 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:55.219 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:55.219 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:55.219 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:55.219 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:55.219 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:55.219 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:55.220 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:55.220 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.220 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:55.220 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.220 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.220 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:55.220 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:56.005 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:57.000 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:57.006 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:58.008 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:59.009 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:59.219 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:59.221 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:59.221 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:59.221 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:59.221 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:59.221 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:59.221 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:59.222 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:59.222 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.222 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:59.222 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.222 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.222 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:59.222 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:00.010 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:01.011 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:02.013 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:03.014 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:03.222 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:03.223 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:03.224 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:03.224 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:03.224 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:03.224 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:03.224 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:03.224 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:03.224 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.225 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:03.225 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.225 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.225 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:03.225 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:04.016 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:05.017 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:06.019 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:07.020 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:07.224 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:07.226 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:07.226 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:07.226 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:07.226 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:07.226 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:07.226 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:07.226 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:07.226 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.227 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:07.227 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.227 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.227 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:07.227 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:08.022 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:09.023 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:09.318 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:10.025 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:11.026 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:11.226 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:11.228 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:11.228 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:11.228 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:11.228 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:11.228 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:11.228 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:11.229 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:11.229 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.229 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:11.229 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.229 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.229 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:11.229 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:12.028 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:13.030 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:14.032 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:14.705 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:12:14.706 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.706 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:12:14.706 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.033 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:15.044 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=22) 09/01/2025 02:12:15.044 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=22) is either full or getting a new batch 09/01/2025 02:12:15.044 7f1bc34eca40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:12:15.044 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:12:15.044 7f1bc34eca40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=22) 09/01/2025 02:12:15.044 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Node 2 not ready; delaying produce of accumulated batch (0.000000 ms) 09/01/2025 02:12:15.044 7f1bc33e6b80 [kafka.client] [DEBUG]: Initiating connection to node 2 at 10.0.0.38:9092 09/01/2025 02:12:15.045 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-2.bytes-sent 09/01/2025 02:12:15.045 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-2.bytes-received 09/01/2025 02:12:15.045 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-2.latency 09/01/2025 02:12:15.045 7f1bc33e6b80 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-2.throttle 09/01/2025 02:12:15.046 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:12:15.046 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:12:15.046 7f1bc33e6b80 [kafka.conn] [INFO]: [IPv4 ('10.0.0.38', 9092)]>: connecting to 10.0.0.38:9092 [('10.0.0.38', 9092) IPv4] 09/01/2025 02:12:15.046 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 239913.908447, idle connection 59649.000000, request inf 09/01/2025 02:12:15.047 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Node 2 not ready; delaying produce of accumulated batch (48.985529 ms) 09/01/2025 02:12:15.047 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 48.974752, metadata 239913.523193, idle connection 59648.000000, request inf 09/01/2025 02:12:15.047 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: established TCP connection 09/01/2025 02:12:15.047 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: checking broker Api Versions 09/01/2025 02:12:15.047 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: Using pre-configured api_version (2, 6) for ApiVersions 09/01/2025 02:12:15.047 7f1bc33e6b80 [kafka.conn] [INFO]: [IPv4 ('10.0.0.38', 9092)]>: Connection complete. 09/01/2025 02:12:15.047 7f1bc33e6b80 [kafka.client] [DEBUG]: Node 2 connected 09/01/2025 02:12:15.047 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {2} 09/01/2025 02:12:15.048 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {2: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xf3`\xa4\xdb\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\xe4D\x00\x00\x01\x99\x03\x0b\xe4D\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:12:15.048 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xf3`\xa4\xdb\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\xe4D\x00\x00\x01\x99\x03\x0b\xe4D\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:12:15.048 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xf3`\xa4\xdb\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\xe4D\x00\x00\x01\x99\x03\x0b\xe4D\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:12:15.048 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: Request 1 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xf3`\xa4\xdb\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\xe4D\x00\x00\x01\x99\x03\x0b\xe4D\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:12:15.048 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 239912.036377, idle connection 59647.000000, request 29999.960899 09/01/2025 02:12:15.048 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992380, metadata 239911.677979, idle connection 59646.000000, request 29999.607325 09/01/2025 02:12:15.051 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:12:15.051 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:12:15.051 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: Response 1 (2.9115676879882812 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:12:15.051 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:12:15.051 7f1bc33e6b80 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=22) with base offset 0 log start offset 0. 09/01/2025 02:12:15.051 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991903, metadata 239908.706543, idle connection 59643.000000, request inf 09/01/2025 02:12:15.052 7f1bc34eca40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:12:15.229 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:15.230 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:15.230 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:15.230 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:15.230 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:15.230 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:15.230 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:15.231 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:15.231 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.231 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:15.231 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.231 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.231 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:15.231 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:16.034 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:17.036 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:18.038 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:19.039 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:19.231 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:19.233 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:19.233 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:19.233 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:19.233 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:19.233 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:19.233 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:19.234 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:19.234 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.234 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:19.234 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.234 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.234 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:19.234 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:20.040 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:21.042 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:21.875 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:22.043 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:23.065 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:23.233 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:23.235 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:23.235 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:23.235 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:23.235 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:23.235 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:23.235 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:23.235 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:23.236 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.236 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:23.236 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.236 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.236 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:23.236 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:24.067 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:25.068 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:26.070 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:27.071 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:27.235 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:27.236 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:27.236 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:27.236 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:27.237 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:27.237 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:27.237 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:27.237 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:27.237 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.237 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:27.237 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.237 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.238 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:27.238 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:28.072 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:29.073 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:30.074 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:31.076 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:31.237 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:31.238 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:31.238 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:31.238 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:31.238 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:31.238 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:31.238 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:31.239 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:31.239 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.239 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:31.239 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.239 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.239 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:31.239 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:32.078 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:33.080 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:33.927 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:34.081 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:35.082 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:35.239 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:35.241 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:35.241 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:35.241 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:35.241 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:35.241 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:35.241 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:35.242 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:35.242 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.242 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:35.242 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.242 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.242 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:35.242 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:36.083 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:37.084 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:38.086 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:39.088 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:39.242 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:39.244 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:39.244 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:39.244 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:39.244 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:39.244 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:39.244 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:39.245 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:39.245 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.245 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:39.245 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.245 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.245 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:39.245 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:40.089 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:41.091 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:42.092 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:43.094 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:43.244 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:43.245 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:43.245 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:43.245 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:43.245 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:43.246 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:43.246 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:43.246 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:43.246 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.246 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:43.246 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.246 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.246 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:43.246 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.096 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:45.098 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:46.045 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:46.098 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:47.100 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:47.246 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:47.248 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:47.248 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:47.248 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:47.248 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:47.248 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:47.248 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:47.249 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:47.249 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.249 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:47.249 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.249 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.249 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:47.249 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:48.102 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:49.104 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:50.106 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:51.107 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:51.248 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:51.250 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:51.250 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:51.250 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:51.250 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:51.250 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:51.250 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:51.251 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:51.251 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.251 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:51.251 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.251 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.251 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:51.251 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.109 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:53.110 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:54.112 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:55.114 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:55.251 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:55.253 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:55.253 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:55.253 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:55.253 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:55.253 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:55.253 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:55.254 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:55.254 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.254 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:55.254 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.254 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.254 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:55.254 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:56.116 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:57.117 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:58.119 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:58.322 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:59.121 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:59.253 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:59.255 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:59.255 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:59.255 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:59.255 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:59.256 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:59.256 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:59.256 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:59.256 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.256 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:59.256 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.256 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.256 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:59.256 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.123 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:01.125 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:02.127 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:03.129 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:03.256 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:03.258 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:03.258 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:03.258 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:03.258 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:03.258 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:03.258 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:03.259 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:03.259 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.259 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:03.259 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.259 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.259 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:03.259 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.131 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:05.132 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:06.134 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:07.136 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:07.259 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:07.261 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:07.261 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:07.261 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:07.261 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:07.261 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:07.261 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:07.262 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:07.262 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.262 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:07.262 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.262 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.262 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:07.262 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.138 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:09.140 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:10.142 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:10.550 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:11.144 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:11.261 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:11.262 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:11.262 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:11.262 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:11.263 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:11.263 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:11.263 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:11.263 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:11.263 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.263 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:11.263 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.263 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.263 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:11.263 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.146 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:13.148 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:14.150 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:14.696 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.985704, metadata 180264.347412, idle connection 120253.000000, request inf 09/01/2025 02:13:14.705 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:13:14.706 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.706 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:13:14.706 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.151 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:15.263 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:15.264 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:15.264 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:15.264 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:15.264 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:15.264 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:15.265 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:15.265 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:15.265 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.265 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:15.265 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.265 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.265 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:15.265 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.153 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:17.155 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:18.156 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:19.159 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:19.264 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:19.266 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:19.266 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:19.266 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:19.266 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:19.266 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:19.266 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:19.267 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:19.267 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.267 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:19.267 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.267 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.267 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:19.267 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.160 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:21.162 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:22.164 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:23.112 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:23.165 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:23.267 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:23.268 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:23.268 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:23.269 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:23.269 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:23.269 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:23.269 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:23.269 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:23.269 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.269 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:23.269 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.270 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.270 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:23.270 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.167 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:25.168 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:26.170 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:27.171 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:27.269 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:27.271 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:27.271 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:27.271 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:27.271 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:27.271 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:27.271 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:27.271 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:27.271 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.272 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:27.272 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.272 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.272 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:27.272 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:28.173 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:29.175 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:30.177 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:31.179 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:31.271 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:31.273 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:31.273 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:31.273 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:31.273 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:31.273 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:31.274 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:31.274 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:31.274 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.274 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:31.274 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.274 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.274 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:31.274 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.182 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:33.184 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:34.186 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:35.187 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:35.274 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:35.276 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:35.276 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:35.276 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:35.276 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:35.276 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:35.276 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:35.276 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:35.277 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.277 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:35.277 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.277 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.277 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:35.277 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.297 7f1bc3452860 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:36.188 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:37.191 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:38.192 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:39.194 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:39.276 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:39.278 7f1bc3452860 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:39.278 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:39.278 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:39.278 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:39.278 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:39.278 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:39.278 7f1bc3452860 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:39.278 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.279 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:39.279 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.279 7f1bc3452860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.279 7f1bc3452860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:39.279 7f1bc3452860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.195 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:41.197 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:42.199 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:43.200 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:43.278 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:43.285 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:43.285 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:43.285 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:43.285 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:43.285 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:43.285 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:43.286 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:43.286 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.286 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:43.286 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.286 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.286 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:43.286 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.202 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:45.203 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:46.205 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:47.250 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:47.285 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:47.287 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:47.287 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:47.287 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:47.288 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:47.288 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:47.288 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:47.288 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:47.288 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.288 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:47.288 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.288 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.289 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:47.289 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.573 7f1bc2ddd220 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:48.251 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:49.254 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:50.256 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:51.257 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:51.288 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:51.289 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:51.289 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:51.289 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:51.289 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:51.289 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:51.290 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:51.290 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:51.290 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.290 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:51.290 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.290 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.290 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:51.290 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.260 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:53.261 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:54.263 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:55.264 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:55.290 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:55.291 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:55.291 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:55.291 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:55.292 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:55.292 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:55.292 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:55.292 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:55.292 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.292 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:55.292 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.292 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.292 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:55.292 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:56.267 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:57.268 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:58.270 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:59.271 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:59.292 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:59.293 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:59.293 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:59.293 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:59.294 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:59.294 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:59.294 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:59.294 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:59.294 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.294 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:59.294 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.294 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.294 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:59.295 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.626 7f1bc2ddd220 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:00.273 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:01.275 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:02.276 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:03.278 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:03.293 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:03.294 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:03.295 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:03.295 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:03.295 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:03.295 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:03.295 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:03.295 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:03.295 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.295 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:03.295 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.295 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.296 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:03.296 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.279 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:05.281 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:06.282 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:07.283 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:07.295 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:07.296 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:07.296 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:07.296 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:07.297 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:07.297 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:07.297 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:07.297 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:07.297 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.297 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:07.297 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.297 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.297 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:07.297 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.285 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:09.287 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:10.288 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:11.289 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:11.297 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:11.298 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:11.298 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:11.298 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:11.299 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:11.299 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:11.299 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:11.299 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:11.299 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.299 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:11.299 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.299 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.299 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:11.299 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.597 7f1bc2ddd220 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:12.291 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:13.293 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:14.294 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:14.706 7f1bc34ec2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:14:14.707 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.707 7f1bc2ddd220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:14:14.707 7f1bc2ddd220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.053 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=19) 09/01/2025 02:14:15.053 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=19) is either full or getting a new batch 09/01/2025 02:14:15.053 7f1bc34eca40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:14:15.053 7f1bc34eca40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:14:15.054 7f1bc34eca40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=19) 09/01/2025 02:14:15.054 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {2} 09/01/2025 02:14:15.054 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {2: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=19, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xafkAx\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\r\xb9\r\x00\x00\x01\x99\x03\r\xb9\r\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:14:15.054 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=19, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xafkAx\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\r\xb9\r\x00\x00\x01\x99\x03\r\xb9\r\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:14:15.054 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=19, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xafkAx\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\r\xb9\r\x00\x00\x01\x99\x03\r\xb9\r\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:14:15.055 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: Request 2 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=19, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xafkAx\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\r\xb9\r\x00\x00\x01\x99\x03\r\xb9\r\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:14:15.055 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 119905.378174, idle connection 59894.000000, request 29999.958515 09/01/2025 02:14:15.055 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992380, metadata 119904.953613, idle connection 59894.000000, request 29999.540567 09/01/2025 02:14:15.058 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:14:15.058 7f1bc33e6b80 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:14:15.058 7f1bc33e6b80 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: Response 2 (3.267049789428711 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=19, error_code=0, offset=1, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:14:15.058 7f1bc33e6b80 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=19, error_code=0, offset=1, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:14:15.058 7f1bc33e6b80 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=19) with base offset 1 log start offset 0. 09/01/2025 02:14:15.058 7f1bc33e6b80 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992380, metadata 119901.697998, idle connection 59890.000000, request inf 09/01/2025 02:14:15.059 7f1bc34eca40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:14:15.296 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:15.299 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:15.299 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:15.300 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:15.300 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:15.300 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:15.300 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:15.300 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:15.300 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:15.300 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.300 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:15.300 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.300 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.301 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:15.301 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.297 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:17.299 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:18.300 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:19.300 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:19.301 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:19.302 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:19.302 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:19.302 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:19.302 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:19.302 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:19.302 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:19.303 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:19.303 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.303 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:19.303 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.303 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.303 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:19.303 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:20.302 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:21.304 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:22.306 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:23.304 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:23.306 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:23.306 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:23.306 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:23.306 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:23.306 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:23.306 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:23.306 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:23.306 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.307 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:23.307 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.307 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.307 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:23.307 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.307 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:24.164 7f1bc2ddd360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:24.308 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:25.310 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:26.174 7f1bc2ddd360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:26.311 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:27.307 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:27.308 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:27.308 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:27.308 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:27.308 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:27.308 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:27.308 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:27.309 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:27.309 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.309 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:27.309 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.309 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.309 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:27.309 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.312 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:28.173 7f1bc2ddd360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:28.313 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:29.315 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:30.316 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:31.309 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:31.311 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:31.311 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:31.311 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:31.311 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:31.311 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:31.311 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:31.312 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:31.312 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.312 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:31.312 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.312 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.312 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:31.312 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.317 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:32.319 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:33.320 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:34.321 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:35.311 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:35.313 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:35.313 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:35.313 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:35.313 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:35.313 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:35.313 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:35.313 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:35.313 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.313 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:35.314 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.314 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.314 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:35.314 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.323 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:36.324 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:37.326 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:38.328 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:39.314 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:39.315 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:39.315 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:39.315 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:39.315 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:39.315 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:39.315 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:39.316 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:39.316 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.316 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:39.316 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.316 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.316 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:39.316 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.329 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:40.330 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:41.331 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:42.333 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:43.316 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:43.318 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:43.318 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:43.318 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:43.318 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:43.318 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:43.318 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:43.319 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:43.319 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.319 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:43.319 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.319 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.319 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:43.319 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.335 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:44.335 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:45.336 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:45.775 7f1bc2ddd360 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:46.338 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:47.318 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:47.320 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:47.320 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:47.320 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:47.320 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:47.320 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:47.320 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:47.320 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:47.320 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.321 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:47.321 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.321 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.321 7f1bc2ddd360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954691383 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:47.321 7f1bc2ddd360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.339 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:48.341 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:49.342 7f1bc34ec540 [contrail-alarm-gen] [INFO]: UVE Done