09/01/2025 01:59:14.316 7f1b5f467600 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:59:14.316 7f1b5f467600 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:59:14.316 7f1b5f467600 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:59:14.322 7f1b5f467600 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/01/2025 01:59:14.323 7f1b5b01ad60 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/01/2025 01:59:14.323 7f1b5b01ad60 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/01/2025 01:59:14.323 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:59:14.323 7f1b5b03ec20 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 01:59:14.324 7f1b5b03ec20 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:59:14.324 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:59:14.324 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.324 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:14.324 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.324 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.324 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/01/2025 01:59:14.324 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.325 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.325 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:59:14.325 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.325 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.326 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:59:14.326 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.326 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.326 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:59:14.326 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.326 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.326 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:59:14.326 7f1b5b03ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.326 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:59:14.326 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/01/2025 01:59:14.327 7f1b5afe6d60 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/01/2025 01:59:14.328 7f1b5af81ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.328 7f1b5af81ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 01:59:14.328 7f1b5af81ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.328 7f1b5af81ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.328 7f1b5af81ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:59:14.328 7f1b5af81ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.382 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.131:5673//'] 09/01/2025 01:59:14.390 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:59:14.391 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.391 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.391 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.391 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.391 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.249:6379[Redis Instance initialized], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.391 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.393 7f1b5af93720 [kazoo.client] [INFO]: Connecting to 10.0.0.131(10.0.0.131):2181, use_ssl: False 09/01/2025 01:59:14.394 7f1b5af930e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.394 7f1b5af930e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.394 7f1b5af930e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.395 7f1b5af93720 [kazoo.client] [DEBUG]: Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=60000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) 09/01/2025 01:59:14.396 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:59:14.396 7f1b5af930e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.396 7f1b5af930e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:59:14.396 7f1b5af930e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.403 7f1b5ae95360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.403 7f1b5ae95360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.131:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 01:59:14.403 7f1b5ae95360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.404 7f1b5af93720 [kazoo.client] [INFO]: Zookeeper connection established, state: CONNECTED 09/01/2025 01:59:14.404 7f1b5af93720 [contrail-alarm-gen] [INFO]: Analytics Discovery listen CONNECTED 09/01/2025 01:59:14.404 7f1b5af93720 [contrail-alarm-gen] [INFO]: Analytics Discovery to publish None 09/01/2025 01:59:14.404 7f1b5ae95360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.405 7f1b5ae95360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper re-established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.131:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:59:14.405 7f1b5ae95360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.405 7f1b5ae95360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.405 7f1b5ae95360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.131:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 01:59:14.405 7f1b5ae95360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.405 7f1b5af93720 [kazoo.client] [DEBUG]: Sending request(xid=1): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:14.406 7f1b5af93720 [kazoo.client] [DEBUG]: Received response(xid=1): ZnodeStat(czxid=4294967571, mzxid=4294967571, ctime=1756691938781, mtime=1756691938781, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=4294967571) 09/01/2025 01:59:14.406 7f1b5ae95360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.406 7f1b5ae95360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:14.407 7f1b5ae95360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.407 7f1b5af93720 [kazoo.client] [DEBUG]: Sending request(xid=2): GetChildren(path='/analytics-discovery-/AlarmGenerator', watcher=>) 09/01/2025 01:59:14.408 7f1b5af93720 [kazoo.client] [DEBUG]: Received response(xid=2): [] 09/01/2025 01:59:14.408 7f1b5afe6d60 [contrail-alarm-gen] [INFO]: Analytics Discovery Watcher AlarmGenerator Children [] 09/01/2025 01:59:14.408 7f1b5af93720 [kazoo.client] [DEBUG]: Sending request(xid=3): Exists(path='/analytics-discovery-/Collector', watcher=None) 09/01/2025 01:59:14.409 7f1b5af93720 [kazoo.client] [DEBUG]: Received response(xid=3): ZnodeStat(czxid=4294967572, mzxid=4294967572, ctime=1756691938785, mtime=1756691938785, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=4294967572) 09/01/2025 01:59:14.409 7f1b5af93720 [kazoo.client] [DEBUG]: Sending request(xid=4): GetChildren(path='/analytics-discovery-/Collector', watcher=>) 09/01/2025 01:59:14.410 7f1b5af93720 [kazoo.client] [DEBUG]: Received response(xid=4): [] 09/01/2025 01:59:14.410 7f1b5afe6d60 [contrail-alarm-gen] [INFO]: Analytics Discovery AlarmGenerator reconnect 09/01/2025 01:59:14.411 7f1b5afe6d60 [contrail-alarm-gen] [INFO]: ensure /analytics-discovery-/AlarmGenerator 09/01/2025 01:59:14.411 7f1b5afe6d60 [contrail-alarm-gen] [INFO]: zk state CONNECTED (CONNECTED) 09/01/2025 01:59:14.411 7f1b5af93720 [kazoo.client] [DEBUG]: Sending request(xid=5): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:14.411 7f1b5af93720 [kazoo.client] [DEBUG]: Received response(xid=5): ZnodeStat(czxid=4294967571, mzxid=4294967571, ctime=1756691938781, mtime=1756691938781, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=4294967571) 09/01/2025 01:59:14.412 7f1b5afe6d60 [contrail-alarm-gen] [INFO]: check for /analytics-discovery-/AlarmGenerator/cn-jenkins-deploy-platform-ansible-os-3995-1.-0 09/01/2025 01:59:14.412 7f1b5af93720 [kazoo.client] [DEBUG]: Sending request(xid=6): Exists(path='/analytics-discovery-/AlarmGenerator/cn-jenkins-deploy-platform-ansible-os-3995-1.-0', watcher=None) 09/01/2025 01:59:14.413 7f1b5af93720 [kazoo.client] [DEBUG]: Sending request(xid=7): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:14.414 7f1b5af93720 [kazoo.client] [DEBUG]: Received response(xid=7): ZnodeStat(czxid=4294967571, mzxid=4294967571, ctime=1756691938781, mtime=1756691938781, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=4294967571) 09/01/2025 01:59:14.414 7f1b5af93720 [kazoo.client] [DEBUG]: Sending request(xid=8): GetChildren(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 01:59:14.414 7f1b5af93720 [kazoo.client] [DEBUG]: Received response(xid=8): [] 09/01/2025 01:59:14.415 7f1b5afe6d60 [contrail-alarm-gen] [INFO]: Discovery AG callback : dict_values([]) 09/01/2025 01:59:14.439 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:59:14.440 7f1b5aee6e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.440 7f1b5aee6e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:14.440 7f1b5aee6e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.442 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:59:14.443 7f1b5aee6e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.443 7f1b5aee6e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:14.443 7f1b5aee6e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.443 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/01/2025 01:59:14.443 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/01/2025 01:59:14.443 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:59:14.444 7f1b5aee6e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:59:14.444 7f1b5ae95ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.445 7f1b5ae95ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:14.445 7f1b5ae95ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.445 7f1b5ae95ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.445 7f1b5ae95ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:14.445 7f1b5ae95ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.450 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:59:14.453 7f1b5aee6e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:59:14.454 7f1b5aeaa220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.454 7f1b5aeaa220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:14.454 7f1b5aeaa220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.478 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:59:14.480 7f1b5ae5d9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.480 7f1b5ae5d9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:59:14.480 7f1b5ae5d9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.500 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:59:14.500 7f1b5ae5d220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.501 7f1b5ae5d220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:59:14.501 7f1b5ae5d220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.510 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:59:14.551 7f1b5b03ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:59:14.552 7f1b5a7a6a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.552 7f1b5a7a6a40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:59:14.552 7f1b5a7a6a40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.604 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config config_type = global_system_config operation = UPDATE config = { uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"fq_name : ["default-global-system-config"]autonomous_system : 64512enable_4byte_as : falseconfig_version : "1.0"ibgp_auto_mesh : truemac_aging_time : 300igmp_enable : falseenable_security_policy_draft : falseid_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16062397226589833301, "uuid_lslong": 9453937659331017976}, "enable": true, "created": "2025-09-01T01:56:54.837560", "last_modified": "2025-09-01T01:57:57.527758", "description": null, "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"], "attr": null, "uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": null, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": null, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}] } 09/01/2025 01:59:14.604 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-address-mismatch-compute config_type = alarm operation = UPDATE config = { uuid : "fa3f6a2e-8282-4f6f-80ce-33639a52499d"fq_name : ["default-global-system-config", "system-defined-address-mismatch-compute"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_ip_address", "operand2": {"uve_attribute": "VrouterAgent.control_ip", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18032248181005373295, "uuid_lslong": 9281412384941033885}, "enable": true, "created": "2025-09-01T01:57:49.827148", "last_modified": "2025-09-01T01:57:50.038909", "description": "Compute Node IP Address mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-address-mismatch-compute" } 09/01/2025 01:59:14.605 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-address-mismatch-control config_type = alarm operation = UPDATE config = { uuid : "c8266300-cc83-4a49-bd56-9c8537aad9c1"fq_name : ["default-global-system-config", "system-defined-address-mismatch-control"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "not in", "operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14422323711782898249, "uuid_lslong": 13643264217144023489}, "enable": true, "created": "2025-09-01T01:57:49.990142", "last_modified": "2025-09-01T01:57:50.257402", "description": "Control Node IP Address mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-address-mismatch-control" } 09/01/2025 01:59:14.605 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-bgp-connectivity config_type = alarm operation = UPDATE config = { uuid : "81d55dd2-bdf1-491a-b083-a3fc512867ea"fq_name : ["default-global-system-config", "system-defined-bgp-connectivity"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "BgpRouterState.num_up_bgp_peer", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "BgpRouterState.num_up_bgp_peer", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9355486960643295514, "uuid_lslong": 12719190076713101290}, "enable": true, "created": "2025-09-01T01:57:50.146892", "last_modified": "2025-09-01T01:57:50.443758", "description": "BGP peer mismatch. Not enough BGP peers are up.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-bgp-connectivity" } 09/01/2025 01:59:14.605 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-bottle-request-size-limit config_type = alarm operation = UPDATE config = { uuid : "b9cad561-1c37-4571-bb68-f9d9eca108c7"fq_name : ["default-global-system-config", "system-defined-bottle-request-size-limit"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["config-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "UveConfigReq.err_info.*.", "operand2": {"uve_attribute": null, "json_value": "True"}, "variables": []}]}]}id_perms : {"permissions": null, "uuid": {"uuid_mslong": 13387747455374148977, "uuid_lslong": 13504318197043628231}, "enable": null, "created": "2025-09-01T01:56:56.131127", "last_modified": "2025-09-01T01:56:56.131127", "description": "Bottle request size limit exceeded.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []} } 09/01/2025 01:59:14.605 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-conf-elements-inconsistency config_type = alarm operation = UPDATE config = { uuid : "03bc6282-7e14-4f38-badd-7f9f072b4620"fq_name : ["default-global-system-config", "system-defined-conf-elements-inconsistency"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "size==", "operand1": "ContrailConfig.elements", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 269198390335917880, "uuid_lslong": 13465058781904455200}, "enable": true, "created": "2025-09-01T01:57:49.704740", "last_modified": "2025-09-01T01:57:49.826527", "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-conf-elements-inconsistency" } 09/01/2025 01:59:14.606 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-conf-incorrect config_type = alarm operation = UPDATE config = { uuid : "b17d423d-1f4f-42e4-ab55-b705b7d714e6"fq_name : ["default-global-system-config", "system-defined-conf-incorrect"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "ContrailConfig", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12789451347087803108, "uuid_lslong": 12345975188693128422}, "enable": true, "created": "2025-09-01T01:57:50.244269", "last_modified": "2025-09-01T01:57:50.614160", "description": "ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-conf-incorrect" } 09/01/2025 01:59:14.606 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-core-files config_type = alarm operation = UPDATE config = { uuid : "29f93096-c40c-408a-94a3-070ccbe9c499"fq_name : ["default-global-system-config", "system-defined-core-files"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "NodeStatus.all_core_file_list", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "NodeStatus.all_core_file_list", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3024502048848363658, "uuid_lslong": 10710412090359268505}, "enable": true, "created": "2025-09-01T01:57:51.398898", "last_modified": "2025-09-01T01:57:52.970710", "description": "A core file has been generated on the node.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-core-files" } 09/01/2025 01:59:14.606 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-disk-usage-critical config_type = alarm operation = UPDATE config = { uuid : "149fff35-9ae3-4f8d-906f-8423cd373cb2"fq_name : ["default-global-system-config", "system-defined-disk-usage-critical"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": ">", "operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operand2": {"uve_attribute": null, "json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1486187007752490893, "uuid_lslong": 10407682553178176690}, "enable": true, "created": "2025-09-01T01:57:50.437138", "last_modified": "2025-09-01T01:57:50.956619", "description": "Disk usage crosses critical threshold limit.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-disk-usage-critical" } 09/01/2025 01:59:14.606 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-disk-usage-high config_type = alarm operation = UPDATE config = { uuid : "04d7992e-6679-4bd3-8298-737ed5150d31"fq_name : ["default-global-system-config", "system-defined-disk-usage-high"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "range", "operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operand2": {"uve_attribute": null, "json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 348915920711273427, "uuid_lslong": 9410398409968651569}, "enable": true, "created": "2025-09-01T01:57:50.343229", "last_modified": "2025-09-01T01:57:50.782298", "description": "Disk usage crosses high threshold limit.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-disk-usage-high" } 09/01/2025 01:59:14.607 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-node-status config_type = alarm operation = UPDATE config = { uuid : "64598318-cd07-4a87-af8d-efa93bf77779"fq_name : ["default-global-system-config", "system-defined-node-status"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7230954819262302855, "uuid_lslong": 12650030438488176505}, "enable": true, "created": "2025-09-01T01:57:50.537284", "last_modified": "2025-09-01T01:57:51.124011", "description": "Node Failure. NodeStatus UVE not present.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-node-status" } 09/01/2025 01:59:14.607 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-package-version-mismatch config_type = alarm operation = UPDATE config = { uuid : "f72985ec-b0f3-47a5-94cf-753bc9704e58"fq_name : ["default-global-system-config", "system-defined-package-version-mismatch"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "NodeStatus.running_package_version", "operand2": {"uve_attribute": "NodeStatus.installed_package_version", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17809913453040846757, "uuid_lslong": 10722918137435541080}, "enable": true, "created": "2025-09-01T01:57:51.577756", "last_modified": "2025-09-01T01:57:53.321232", "description": "There is a mismatch between installed and running package version.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-package-version-mismatch" } 09/01/2025 01:59:14.607 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-partial-sysinfo config_type = alarm operation = UPDATE config = { uuid : "d7676398-52ea-482c-a5c5-2fc5683b53c4"fq_name : ["default-global-system-config", "system-defined-partial-sysinfo"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.build_info", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15521484146632968236, "uuid_lslong": 11945006111573889988}, "enable": true, "created": "2025-09-01T01:57:50.636982", "last_modified": "2025-09-01T01:57:51.305614", "description": "System Info Incomplete.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-partial-sysinfo" } 09/01/2025 01:59:14.607 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks config_type = alarm operation = UPDATE config = { uuid : "d08b3c4f-8364-42cc-b19f-c64aec697ac0"fq_name : ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["database-nodeconfig-database-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": ">=", "operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"uve_attribute": null, "json_value": "300"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15027170893856260812, "uuid_lslong": 12799166691106454208}, "enable": true, "created": "2025-09-01T01:57:51.495240", "last_modified": "2025-09-01T01:57:53.136268", "description": "Pending compaction tasks in cassandra crossed the configured threshold.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-pending-cassandra-compaction-tasks" } 09/01/2025 01:59:14.608 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-process-connectivity config_type = alarm operation = UPDATE config = { uuid : "6177a338-54f3-47b2-b8cd-742fd97fe3cd"fq_name : ["default-global-system-config", "system-defined-process-connectivity"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.process_status", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "NodeStatus.process_status.state", "operand2": {"uve_attribute": null, "json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7023261606246303666, "uuid_lslong": 13316427422065746893}, "enable": true, "created": "2025-09-01T01:57:50.713917", "last_modified": "2025-09-01T01:57:51.470711", "description": "Process(es) reporting as non-functional.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-process-connectivity" } 09/01/2025 01:59:14.608 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-process-status config_type = alarm operation = UPDATE config = { uuid : "810dc9aa-8015-43ae-903f-6caad1a28708"fq_name : ["default-global-system-config", "system-defined-process-status"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.process_info", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "NodeStatus.process_info.process_state", "operand2": {"uve_attribute": null, "json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9299310539720442798, "uuid_lslong": 10394145945911723784}, "enable": true, "created": "2025-09-01T01:57:50.802208", "last_modified": "2025-09-01T01:57:51.647167", "description": "Process Failure.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-process-status" } 09/01/2025 01:59:14.608 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-prouter-connectivity config_type = alarm operation = UPDATE config = { uuid : "2b30d6f6-0211-4eb5-99c7-79101d9a1ff9"fq_name : ["default-global-system-config", "system-defined-prouter-connectivity"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["prouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.connected_agent_list", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3112223694598000309, "uuid_lslong": 11080958518291472377}, "enable": true, "created": "2025-09-01T01:57:50.885227", "last_modified": "2025-09-01T01:57:51.816652", "description": "Prouter connectivity to controlling tor agent does not exist we look for non-empty value for connected_agent_list", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-prouter-connectivity" } 09/01/2025 01:59:14.608 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-prouter-tsn-connectivity config_type = alarm operation = UPDATE config = { uuid : "935eb4db-73bc-4a29-8c4d-e2ed2941efd3"fq_name : ["default-global-system-config", "system-defined-prouter-tsn-connectivity"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["prouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.tsn_agent_list", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}, {"operation": "!=", "operand1": "ProuterData.gateway_mode", "operand2": {"uve_attribute": null, "json_value": "\"SERVER\""}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.tsn_agent_list", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": []}, {"operation": "==", "operand1": "ProuterData.gateway_mode", "operand2": {"uve_attribute": null, "json_value": "\"SERVER\""}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10619123826018765353, "uuid_lslong": 10109986246743945171}, "enable": true, "created": "2025-09-01T01:57:50.981229", "last_modified": "2025-09-01T01:57:51.990045", "description": "Prouter connectivity to controlling tsn agent does not exist we look for non-empty value for tsn_agent_list", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-prouter-tsn-connectivity" } 09/01/2025 01:59:14.609 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-storage-cluster-state config_type = alarm operation = UPDATE config = { uuid : "8e1480d5-277f-405c-a1ae-7e859515d832"fq_name : ["default-global-system-config", "system-defined-storage-cluster-state"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["storage-cluster"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "StorageCluster.info_stats.status", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10237949505899020380, "uuid_lslong": 11650388398251038770}, "enable": true, "created": "2025-09-01T01:57:51.062109", "last_modified": "2025-09-01T01:57:52.195183", "description": "Storage Cluster warning/errors.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-storage-cluster-state" } 09/01/2025 01:59:14.609 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-interface config_type = alarm operation = UPDATE config = { uuid : "a3cde80b-b0cf-4715-9c86-8fec97c15a7b"fq_name : ["default-global-system-config", "system-defined-vrouter-interface"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": ">=", "operand1": "VrouterAgent.down_interface_count", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11803345335316596501, "uuid_lslong": 11278860563117070971}, "enable": true, "created": "2025-09-01T01:57:51.143961", "last_modified": "2025-09-01T01:57:52.395443", "description": "Vrouter interface(s) down.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-interface" } 09/01/2025 01:59:14.609 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down config_type = alarm operation = UPDATE config = { uuid : "0611e1ec-1ca8-48db-9ad2-92f299cb8e5c"fq_name : ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.l3mh_phy_interface_down", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 437379043040970971, "uuid_lslong": 11156140797610135132}, "enable": true, "created": "2025-09-01T01:57:51.847772", "last_modified": "2025-09-01T01:57:53.866729", "description": "On l3 multihomed compute one or more physical interfaces are Inactive.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-l3mh-physical-interface-link-down" } 09/01/2025 01:59:14.609 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-limit-exceeded config_type = alarm operation = UPDATE config = { uuid : "af6de2d9-8690-4972-9338-5b61b5922f04"fq_name : ["default-global-system-config", "system-defined-vrouter-limit-exceeded"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.res_limit", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12641009152992233842, "uuid_lslong": 10608329397487415044}, "enable": true, "created": "2025-09-01T01:57:51.666562", "last_modified": "2025-09-01T01:57:53.502926", "description": "Agent resource usage exceeded configured watermark for resource.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-limit-exceeded" } 09/01/2025 01:59:14.610 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-table-limit-exceeded config_type = alarm operation = UPDATE config = { uuid : "ca0e784a-4398-4836-afc4-17fc122957e4"fq_name : ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.res_table_limit", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14559706905692358710, "uuid_lslong": 12665274423476574180}, "enable": true, "created": "2025-09-01T01:57:51.751034", "last_modified": "2025-09-01T01:57:53.679612", "description": "Agent resource usage exceeded table size for resource in vrouter.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-table-limit-exceeded" } 09/01/2025 01:59:14.610 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-xmpp-close-reason config_type = alarm operation = UPDATE config = { uuid : "8e0c7107-46a5-4e13-bb2e-4bc845f22ee7"fq_name : ["default-global-system-config", "system-defined-xmpp-close-reason"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["xmpp-peer"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "XmppPeerInfoData.close_reason", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "!=", "operand1": "XmppPeerInfoData.state_info.state", "operand2": {"uve_attribute": null, "json_value": "\"Established\""}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10235680329170243091, "uuid_lslong": 13487801257560256231}, "enable": true, "created": "2025-09-01T01:57:51.312606", "last_modified": "2025-09-01T01:57:52.796846", "description": "XMPP connection closed towards peer, alarm has close reason", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-xmpp-close-reason" } 09/01/2025 01:59:14.610 7f1b5b03ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-xmpp-connectivity config_type = alarm operation = UPDATE config = { uuid : "c634c7c0-0322-4d42-9aa5-5929ed1b749f"fq_name : ["default-global-system-config", "system-defined-xmpp-connectivity"]parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "BgpRouterState.num_up_xmpp_peer", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "BgpRouterState.num_up_xmpp_peer", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14282259945798913346, "uuid_lslong": 11143410889604691103}, "enable": true, "created": "2025-09-01T01:57:51.226178", "last_modified": "2025-09-01T01:57:52.608543", "description": "XMPP peer mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-xmpp-connectivity" } 09/01/2025 01:59:15.328 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:16.330 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:17.331 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:18.325 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:18.326 7f1b5ae95e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:18.326 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:18.326 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:18.326 7f1b5ae95e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:18.326 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:18.327 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:18.327 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:18.327 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.327 7f1b5ae95e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.249:6379 server_addrs = [ 10.0.0.249:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:18.327 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.327 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.327 7f1b5ae95e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:18.327 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.332 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:19.333 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:19.407 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.408 7f1b5ae95e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:19.408 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.408 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.408 7f1b5ae95e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.131:6379 server_addrs = [ 10.0.0.131:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:19.408 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.408 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.408 7f1b5ae95e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:19.408 7f1b5ae95e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:20.335 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:21.336 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:21.873 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:59:21.867078', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:21.867078', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13422430401004267031, 'uuid_mslong': 671617895336398686}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '095210ea-dc52-4f5e-ba46-0d55aa436617'}, 'oper': 'CREATE', 'request-id': 'req-0bcb5258-1518-4af3-bc32-66c2afda7f6a', 'type': 'analytics_snmp_node', 'uuid': '095210ea-dc52-4f5e-ba46-0d55aa436617'} 09/01/2025 01:59:21.873 7f1b5ae95a40 [contrail-alarm-gen] [ERROR]: Error in rabbitmq drainer greenlet: 'AnalyticsLogger' object has no attribute 'warn' 09/01/2025 01:59:21.873 7f1b5ae95a40 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:59:21.874 7f1b5af93c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.874 7f1b5af93c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:21.874 7f1b5af93c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.879 7f1b5ae95a40 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:59:21.879 7f1b5af93c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.879 7f1b5af93c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:21.879 7f1b5af93c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.232 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:59:22.227040', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:22.227040', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10043001507123339914, 'uuid_mslong': 13054484782216790713}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b52ad8be-69c6-42b9-8b5f-e8a91ecca28a'}, 'oper': 'CREATE', 'request-id': 'req-cc52ae74-0d4d-419c-a865-1eba05a95d49', 'type': 'analytics_snmp_node', 'uuid': 'b52ad8be-69c6-42b9-8b5f-e8a91ecca28a'} 09/01/2025 01:59:22.232 7f1b5ae95a40 [contrail-alarm-gen] [ERROR]: Error in rabbitmq drainer greenlet: 'AnalyticsLogger' object has no attribute 'warn' 09/01/2025 01:59:22.233 7f1b5ae95a40 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:59:22.233 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.233 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:22.233 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.236 7f1b5ae95a40 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:59:22.236 7f1b5af93c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.237 7f1b5af93c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:22.237 7f1b5af93c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.307 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:59:22.300301', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:22.300301', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10295412184530401449, 'uuid_mslong': 11790706739729089604}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a3a1014f-585b-4444-8ee0-a6d59b29b8a9'}, 'oper': 'CREATE', 'request-id': 'req-79a7348d-975b-4e66-ac0e-d0c0e9ecf4b1', 'type': 'analytics_snmp_node', 'uuid': 'a3a1014f-585b-4444-8ee0-a6d59b29b8a9'} 09/01/2025 01:59:22.307 7f1b5ae95a40 [contrail-alarm-gen] [ERROR]: Error in rabbitmq drainer greenlet: 'AnalyticsLogger' object has no attribute 'warn' 09/01/2025 01:59:22.307 7f1b5ae95a40 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:59:22.308 7f1b5af93c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.308 7f1b5af93c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:59:22.308 7f1b5af93c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.312 7f1b5ae95a40 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:59:22.319 7f1b5a754540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.319 7f1b5a754540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:22.319 7f1b5a754540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.327 7f1b5a754400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:22.328 7f1b5a754400 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:22.328 7f1b5a754400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:22.328 7f1b5a754400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:22.328 7f1b5a754400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:22.328 7f1b5a754400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:22.328 7f1b5a754400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:22.329 7f1b5a754400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:22.329 7f1b5a754400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.329 7f1b5a754400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:22.329 7f1b5a754400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.329 7f1b5a754400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.329 7f1b5a754400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:22.329 7f1b5a754400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.337 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:23.339 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:24.340 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:25.341 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:26.330 7f1b5a754400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:26.331 7f1b5a754400 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.332 7f1b5a754400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:26.333 7f1b5a754400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.343 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:27.343 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:28.344 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:29.345 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:30.332 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:30.334 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:30.334 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:30.334 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:30.334 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:30.334 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:30.334 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:30.334 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:30.335 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.335 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:30.335 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.335 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.335 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:30.335 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.346 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:31.347 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:32.348 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:33.350 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:34.335 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:34.336 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:34.336 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:34.336 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:34.336 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:34.336 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:34.336 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:34.336 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:34.336 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.337 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:34.337 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.337 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.337 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:34.337 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.351 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:35.353 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:36.354 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:37.355 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:38.337 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:38.339 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:38.339 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:38.339 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:38.339 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:38.339 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:38.339 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:38.339 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:38.339 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.340 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:38.340 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.340 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.340 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:38.340 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.356 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:39.357 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:40.359 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:41.360 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:42.339 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:42.340 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:42.340 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:42.340 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:42.340 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:42.340 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:42.340 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:42.341 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:42.341 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.341 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:42.341 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.341 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.341 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:42.341 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.361 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:43.362 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:44.363 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:45.365 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:46.340 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:46.341 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:46.341 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:46.342 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:46.342 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:46.342 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:46.342 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:46.342 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:46.342 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.342 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:46.343 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.343 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.343 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:46.343 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.367 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:47.369 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:48.370 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:49.371 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:50.342 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:50.344 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:50.344 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:50.344 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:50.344 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:50.344 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:50.344 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:50.344 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:50.344 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.345 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:50.345 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.345 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.345 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:50.345 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.372 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:51.374 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:52.374 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:53.376 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:54.345 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:54.346 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:54.346 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:54.346 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:54.346 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:54.346 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:54.346 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:54.347 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:54.347 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.347 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:54.347 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.347 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.347 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:54.347 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.377 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:55.379 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:56.380 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:57.382 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:58.347 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:58.348 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:58.348 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:58.348 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:58.348 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:58.348 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:58.348 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:58.348 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:58.349 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.349 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:58.349 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.349 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.349 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:58.349 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.383 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 01:59:59.385 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:00.386 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:01.387 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:02.349 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:02.350 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:02.350 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:02.350 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:02.351 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:02.351 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:02.351 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:02.351 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:02.351 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.351 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:02.351 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.351 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.351 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:02.352 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.388 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:03.389 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:04.390 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:05.392 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:06.352 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:06.353 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:06.353 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:06.353 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:06.353 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:06.353 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:06.353 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:06.354 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:06.354 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.354 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:06.354 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.354 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.354 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:06.354 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.393 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:07.394 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:08.396 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:09.398 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:10.354 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:10.355 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:10.355 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:10.355 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:10.355 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:10.355 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:10.355 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:10.356 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:10.356 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.356 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:10.356 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.356 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.356 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:10.356 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.399 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:11.401 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:12.402 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:13.404 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:14.328 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:00:14.329 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.329 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:00:14.329 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.355 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:14.357 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:14.357 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:14.357 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:14.357 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:14.357 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:14.357 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:14.357 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:14.358 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.358 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:14.358 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.358 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.358 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:14.358 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.404 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:15.406 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:16.407 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:17.409 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:18.359 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:18.360 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:18.360 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:18.360 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:18.360 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:18.360 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:18.360 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:18.361 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:18.361 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.361 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:18.361 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.361 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.361 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:18.361 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.410 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:19.411 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:20.413 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:21.415 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:22.361 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:22.362 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:22.362 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:22.362 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:22.362 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:22.362 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:22.362 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:22.363 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:22.363 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.363 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:22.363 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.363 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.363 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:22.363 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.416 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:23.417 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:24.418 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:25.419 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:26.364 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:26.365 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:26.365 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:26.365 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:26.365 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:26.366 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:26.366 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:26.366 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:26.366 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.366 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:26.366 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.366 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.366 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:26.366 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.421 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:27.422 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:28.423 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:29.424 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:30.366 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:30.367 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:30.368 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:30.368 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:30.368 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:30.368 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:30.368 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:30.368 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:30.368 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.368 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:30.368 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.368 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.369 7f1b5af93ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:30.369 7f1b5af93ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.425 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:31.426 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:32.428 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:33.429 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:34.368 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:34.369 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:34.369 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:34.369 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:34.369 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:34.369 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:34.369 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:34.370 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:34.370 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.370 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:34.370 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.370 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.370 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:34.370 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.431 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:35.432 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:36.434 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:37.434 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:38.369 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:38.371 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:38.371 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:38.371 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:38.371 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:38.371 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:38.371 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:38.372 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:38.372 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.372 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:38.372 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.372 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.372 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:38.372 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.436 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:39.436 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:40.437 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:41.439 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:42.372 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:42.374 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:42.374 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:42.374 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:42.374 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:42.374 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:42.374 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:42.375 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:42.375 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.375 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:42.375 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.375 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.375 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:42.375 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.440 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:43.441 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:44.442 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:45.445 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:46.375 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:46.376 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:46.376 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:46.376 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:46.376 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:46.376 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:46.376 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:46.376 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:46.376 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.377 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:46.377 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.377 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.377 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:46.377 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.446 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:47.447 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:48.448 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:49.449 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:50.377 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:50.379 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:50.379 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:50.379 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:50.379 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:50.379 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:50.379 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:50.379 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:50.379 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.379 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:50.380 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.380 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.380 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:50.380 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.450 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:51.452 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:52.453 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:53.455 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:54.380 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:54.382 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:54.382 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:54.383 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:54.383 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:54.383 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:54.383 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:54.383 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:54.383 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.383 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:54.383 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.383 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.384 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:54.384 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.456 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:55.457 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:56.459 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:57.460 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:58.383 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:58.384 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:58.384 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:58.384 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:58.384 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:58.384 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:58.384 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:58.384 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:58.385 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.385 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:58.385 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.385 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.385 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:58.385 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.461 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:59.463 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:00.464 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:01.465 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:02.385 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:02.386 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:02.387 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:02.387 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:02.387 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:02.387 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:02.387 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:02.387 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:02.387 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.387 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:02.387 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.387 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.388 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:02.388 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.466 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:03.468 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:04.469 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:05.470 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:06.387 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:06.389 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:06.389 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:06.389 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:06.389 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:06.389 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:06.389 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:06.389 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:06.390 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.390 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:06.390 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.390 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.390 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:06.390 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.471 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:07.472 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:08.474 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:09.474 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:10.389 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:10.390 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:10.390 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:10.390 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:10.390 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:10.390 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:10.391 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:10.391 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:10.391 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.391 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:10.391 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.391 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.391 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:10.391 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.475 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:10.480 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-d6447694-7a63-4e16-8ab1-f2203b502079', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 02:01:10.601 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3995-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'], 'id_perms': {'created': '2025-09-01T02:01:10.595372', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:10.595372', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11952340681741785153, 'uuid_mslong': 6586627194885784908}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5b686689-4b18-454c-a5df-3e8619ab5841', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.192'}, 'oper': 'CREATE', 'request-id': 'req-681edb8a-0de7-491e-8459-bf06930acfe8', 'type': 'virtual_router', 'uuid': '5b686689-4b18-454c-a5df-3e8619ab5841'} 09/01/2025 02:01:10.940 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:01:10.925113', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:10.925113', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10735908003063543833, 'uuid_mslong': 17648289949859202616}}, 'parent_type': 'virtual-router', 'parent_uuid': '5b686689-4b18-454c-a5df-3e8619ab5841', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:f4:eb:52:34:6a']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'}]}, 'oper': 'CREATE', 'request-id': 'req-1c3ccdca-0f5f-4532-9c6f-20034282c8ff', 'type': 'virtual_machine_interface', 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819'} 09/01/2025 02:01:10.965 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-1c3ccdca-0f5f-4532-9c6f-20034282c8ff', 'type': 'virtual_machine_interface', 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819'} 09/01/2025 02:01:11.477 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:11.478 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-8b202558-6c4f-4e38-a46f-f0bfe3ec33be', 'type': 'access_control_list', 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'} 09/01/2025 02:01:12.479 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:13.480 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:14.330 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:01:14.330 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.330 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:01:14.330 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.391 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:14.392 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:14.392 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:14.392 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:14.392 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:14.393 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:14.393 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:14.393 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:14.393 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.393 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:14.393 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.393 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.393 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:14.393 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.481 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:15.483 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:16.484 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:17.485 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:18.394 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:18.395 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:18.395 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:18.395 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:18.395 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:18.396 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:18.396 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:18.396 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:18.396 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.396 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:18.396 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.396 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.396 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:18.396 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.487 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:19.487 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:20.488 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:21.489 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:22.397 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:22.398 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:22.398 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:22.398 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:22.398 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:22.398 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:22.398 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:22.399 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:22.399 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.399 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:22.399 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.399 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.399 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:22.399 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.491 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:23.492 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:24.493 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:25.494 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:26.399 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:26.401 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:26.401 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:26.401 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:26.401 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:26.401 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:26.401 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:26.401 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:26.401 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.401 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:26.402 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.402 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.402 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:26.402 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.495 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:27.497 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:28.498 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:29.499 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:30.402 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:30.403 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:30.403 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:30.403 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:30.403 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:30.403 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:30.403 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:30.404 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:30.404 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.404 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:30.404 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.404 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.404 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:30.404 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.500 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:31.502 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:32.503 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:33.504 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:34.404 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:34.405 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:34.405 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:34.405 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:34.405 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:34.406 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:34.406 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:34.406 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:34.406 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.406 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:34.406 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.406 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.406 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:34.406 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.506 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:35.507 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:36.508 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:37.510 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:38.407 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:38.408 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:38.408 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:38.408 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:38.408 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:38.408 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:38.408 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:38.409 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:38.409 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.409 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:38.409 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.409 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.409 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:38.409 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.511 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:39.512 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:40.516 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:41.517 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:42.410 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:42.411 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:42.411 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:42.411 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:42.411 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:42.412 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:42.412 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:42.412 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:42.412 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.412 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:42.412 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.412 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.412 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:42.413 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.518 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:43.519 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:44.519 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:45.521 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:46.413 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:46.414 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:46.414 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:46.414 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:46.414 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:46.414 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:46.414 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:46.415 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:46.415 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.415 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:46.415 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.415 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.415 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:46.415 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.523 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:47.523 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:48.525 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:49.525 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:50.414 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:50.416 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:50.416 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:50.416 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:50.416 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:50.416 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:50.416 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:50.417 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:50.417 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.417 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:50.417 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.417 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.417 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:50.417 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.527 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:51.528 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:52.530 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:53.531 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:54.417 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:54.418 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:54.419 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:54.419 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:54.419 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:54.419 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:54.419 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:54.419 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:54.419 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.419 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:54.419 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.420 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.420 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:54.420 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.532 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:55.533 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:56.534 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:57.535 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:58.420 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:58.421 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:58.421 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:58.421 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:58.421 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:58.421 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:58.421 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:58.422 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:58.422 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.422 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:58.422 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.422 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.422 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:58.422 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.536 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:59.537 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:00.539 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:01.540 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:02.422 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:02.423 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:02.423 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:02.423 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:02.423 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:02.423 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:02.423 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:02.424 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:02.424 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.424 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:02.424 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.424 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.424 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:02.424 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.541 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:02.991 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3995-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'], 'id_perms': {'created': '2025-09-01T02:02:02.985199', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:02.985199', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10627036196304171688, 'uuid_mslong': 16146534956283939804}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.18'}, 'oper': 'CREATE', 'request-id': 'req-7f54cc47-40c2-4259-85af-4adeb8e3ee0f', 'type': 'virtual_router', 'uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8'} 09/01/2025 02:02:03.331 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:02:03.316910', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:03.316910', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12779270811540185455, 'uuid_mslong': 492471125198063140}}, 'parent_type': 'virtual-router', 'parent_uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:06:d5:9b:e1:0f']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'}]}, 'oper': 'CREATE', 'request-id': 'req-e0b4c5ea-eae1-45cd-9751-832bb456eea9', 'type': 'virtual_machine_interface', 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f'} 09/01/2025 02:02:03.357 7f1b5ae95a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-e0b4c5ea-eae1-45cd-9751-832bb456eea9', 'type': 'virtual_machine_interface', 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f'} 09/01/2025 02:02:03.542 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:04.543 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:05.544 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:06.423 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:06.424 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:06.425 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:06.425 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:06.425 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:06.425 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:06.425 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:06.425 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:06.426 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.426 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:06.426 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.426 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.426 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:06.426 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.545 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:07.547 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:08.548 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:09.549 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:10.426 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:10.427 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:10.427 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:10.427 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:10.427 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:10.427 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:10.427 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:10.428 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:10.428 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.428 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:10.428 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.428 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.428 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:10.428 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.550 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:11.552 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:12.553 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:13.554 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:14.331 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:02:14.331 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.331 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:02:14.331 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.428 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:14.429 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:14.429 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:14.429 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:14.430 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:14.430 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:14.430 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:14.430 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:14.430 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.430 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:14.430 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.430 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.430 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:14.430 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.556 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:15.557 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:16.558 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:17.559 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:18.430 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:18.431 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:18.431 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:18.431 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:18.431 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:18.431 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:18.431 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:18.432 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:18.432 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.432 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:18.432 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.432 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.432 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:18.432 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.560 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:19.562 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:20.563 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:21.564 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:22.431 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:22.432 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:22.433 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:22.433 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:22.433 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:22.433 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:22.433 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:22.433 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:22.433 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.434 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:22.434 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.434 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.434 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:22.434 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.565 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:23.566 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:24.567 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:25.569 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:26.434 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:26.435 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:26.435 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:26.435 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:26.435 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:26.435 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:26.435 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:26.436 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:26.436 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.436 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:26.436 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.436 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.436 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:26.436 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.570 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:27.571 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:28.573 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:29.574 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:30.436 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:30.437 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:30.437 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:30.438 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:30.438 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:30.438 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:30.438 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:30.438 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:30.438 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.438 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:30.438 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.438 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.438 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:30.439 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.575 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:31.576 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:32.577 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:33.578 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:34.438 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:34.439 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:34.439 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:34.440 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.579 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:35.580 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:36.581 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:37.583 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:38.441 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:38.442 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:38.443 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:38.443 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:38.443 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:38.443 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:38.443 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:38.443 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:38.443 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.444 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:38.444 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.444 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.444 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:38.444 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.584 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:39.585 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:40.587 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:41.588 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:42.443 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:42.444 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:42.445 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:42.445 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:42.445 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:42.445 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:42.445 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:42.445 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:42.445 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.446 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:42.446 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.446 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.446 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:42.446 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.589 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:43.591 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:44.591 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:45.593 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:46.445 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:46.446 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:46.447 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:46.447 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:46.447 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:46.447 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:46.447 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:46.447 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:46.448 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.448 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:46.448 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.448 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.448 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:46.448 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.594 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:47.595 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:48.596 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:49.597 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:50.448 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:50.450 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:50.450 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:50.450 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:50.450 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:50.450 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:50.450 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:50.451 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:50.451 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.451 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:50.451 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.451 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.451 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:50.451 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.599 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:51.600 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:52.601 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:53.603 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:54.450 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:54.451 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:54.451 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:54.451 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:54.452 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:54.452 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:54.452 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:54.452 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:54.452 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.452 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:54.452 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.452 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.452 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:54.453 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.604 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:55.605 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:56.607 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:57.608 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:58.452 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:58.453 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:58.453 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:58.453 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:58.453 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:58.453 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:58.453 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:58.454 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:58.454 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.454 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:58.454 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.454 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.454 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:58.454 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.609 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:59.611 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:00.612 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:01.614 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:02.454 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:02.456 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:02.456 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:02.456 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:02.456 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:02.456 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:02.456 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:02.457 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:02.457 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.457 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:02.457 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.457 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.457 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:02.458 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.615 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:03.616 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:04.617 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:05.619 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:06.456 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:06.457 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:06.457 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:06.457 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:06.458 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:06.458 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:06.458 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:06.458 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:06.458 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.458 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:06.458 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.458 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.458 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:06.458 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.620 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:07.621 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:08.623 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:09.624 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:10.459 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:10.460 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:10.460 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:10.460 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:10.460 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:10.460 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:10.460 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:10.461 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:10.461 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.461 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:10.461 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.461 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.461 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:10.461 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.625 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:11.627 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:12.628 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:13.630 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:14.332 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:03:14.333 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.333 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:03:14.333 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.461 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:14.462 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:14.462 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:14.463 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:14.463 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:14.463 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:14.463 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:14.463 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:14.463 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.463 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:14.463 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.463 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.463 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:14.464 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.631 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:15.632 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:16.633 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:17.634 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:18.463 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:18.464 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:18.465 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:18.465 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:18.465 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:18.465 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:18.465 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:18.465 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:18.465 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.465 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:18.466 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.466 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.466 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:18.466 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.636 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:19.638 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:20.639 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:21.640 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:22.465 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:22.466 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:22.467 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:22.467 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:22.467 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:22.467 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:22.467 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:22.467 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:22.467 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.467 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:22.467 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.467 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.468 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:22.468 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.641 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:23.643 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:24.644 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:25.645 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:26.467 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:26.469 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:26.469 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:26.469 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:26.469 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:26.469 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:26.469 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:26.469 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:26.469 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.470 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:26.470 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.470 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.470 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:26.470 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.646 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:27.648 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:28.649 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:29.651 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:30.469 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:30.471 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:30.471 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:30.471 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:30.471 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:30.472 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:30.472 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:30.472 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:30.472 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.472 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:30.472 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.472 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.472 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:30.472 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.652 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:31.653 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:32.655 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:33.656 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:34.472 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:34.473 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:34.473 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:34.473 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:34.473 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:34.473 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:34.473 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:34.473 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:34.473 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.474 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:34.474 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.474 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.474 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:34.474 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.657 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:35.658 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:36.659 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:37.661 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:38.473 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:38.475 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:38.475 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:38.475 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:38.475 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:38.475 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:38.475 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:38.476 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:38.476 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.476 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:38.476 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.476 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.476 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:38.476 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.662 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:39.663 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:40.665 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:41.666 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:42.475 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:42.476 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:42.476 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:42.477 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.668 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:43.670 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:44.670 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:45.671 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:46.478 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:46.479 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:46.479 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:46.479 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:46.480 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:46.480 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:46.480 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:46.480 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:46.480 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.480 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:46.480 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.480 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.480 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:46.480 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.673 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:47.674 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:48.675 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:49.677 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:50.480 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:50.481 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:50.482 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:50.482 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:50.482 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:50.482 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:50.482 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:50.482 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:50.482 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.482 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:50.482 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.482 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.483 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:50.483 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.681 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:51.682 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:52.683 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:53.685 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:54.482 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:54.483 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:54.483 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:54.483 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:54.484 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:54.484 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:54.484 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:54.484 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:54.484 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.484 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:54.484 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.484 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.484 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:54.484 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.687 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:55.687 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:56.689 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:57.690 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:58.484 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:58.485 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:58.485 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:58.485 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:58.486 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:58.486 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:58.486 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:58.486 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:58.486 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.486 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:58.486 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.486 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.486 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:58.486 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.691 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:59.692 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:00.693 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:01.695 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:02.487 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:02.488 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:02.488 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:02.489 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.696 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:03.697 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:04.698 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:05.699 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:06.489 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:06.491 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:06.491 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:06.491 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:06.491 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:06.491 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:06.491 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:06.492 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:06.492 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.492 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:06.492 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.492 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.492 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:06.492 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.700 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:07.702 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:08.703 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:09.704 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:10.492 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:10.493 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:10.493 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:10.493 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:10.493 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:10.493 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:10.493 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:10.493 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:10.494 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.494 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:10.494 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.494 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.494 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:10.494 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.705 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:11.707 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:12.707 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:13.709 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:14.328 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Starting the Kafka producer 09/01/2025 02:04:14.329 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name connections-closed 09/01/2025 02:04:14.329 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name connections-created 09/01/2025 02:04:14.329 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name select-time 09/01/2025 02:04:14.329 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name io-time 09/01/2025 02:04:14.329 7f1b5af81a40 [kafka.client] [DEBUG]: Attempting to check version with node bootstrap-2 09/01/2025 02:04:14.329 7f1b5af81a40 [kafka.client] [DEBUG]: Initiating connection to node bootstrap-2 at 10.0.0.131:9092 09/01/2025 02:04:14.329 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-sent-received 09/01/2025 02:04:14.330 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-sent 09/01/2025 02:04:14.330 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-received 09/01/2025 02:04:14.330 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name request-latency 09/01/2025 02:04:14.330 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name throttle-time 09/01/2025 02:04:14.330 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-2.bytes-sent 09/01/2025 02:04:14.330 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-2.bytes-received 09/01/2025 02:04:14.330 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-2.latency 09/01/2025 02:04:14.330 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-2.throttle 09/01/2025 02:04:14.331 7f1b5af81a40 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:04:14.331 7f1b5af81a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:04:14.331 7f1b5af81a40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: connecting to 10.0.0.131:9092 [('10.0.0.131', 9092) IPv4] 09/01/2025 02:04:14.331 7f1b5af81a40 [kafka.client] [DEBUG]: Timeouts: user 199.969053, metadata inf, idle connection inf, request inf 09/01/2025 02:04:14.332 7f1b5af81a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: established TCP connection 09/01/2025 02:04:14.332 7f1b5af81a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: checking broker Api Versions 09/01/2025 02:04:14.332 7f1b5af81a40 [kafka.protocol.parser] [DEBUG]: Sending request ApiVersionsRequest_v4(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:14.332 7f1b5af81a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 1 (timeout_ms 1600.0): ApiVersionsRequest_v4(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:14.332 7f1b5af81a40 [kafka.client] [DEBUG]: Timeouts: user 199.972391, metadata inf, idle connection inf, request 1599.718094 09/01/2025 02:04:14.333 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:04:14.333 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.333 7f1b5ae7b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:04:14.333 7f1b5ae7b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.341 7f1b5af81a40 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:04:14.341 7f1b5af81a40 [kafka.protocol.parser] [DEBUG]: Processing response ApiVersionsResponse_v4 09/01/2025 02:04:14.341 7f1b5af81a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 1 (8.67605209350586 ms): ApiVersionsResponse_v0(error_code=35, api_versions=[(api_key=18, min_version=0, max_version=3)]) 09/01/2025 02:04:14.341 7f1b5af81a40 [kafka.client] [DEBUG]: Timeouts: user 199.953794, metadata inf, idle connection 539987.000000, request inf 09/01/2025 02:04:14.341 7f1b5af81a40 [kafka.protocol.parser] [DEBUG]: Sending request ApiVersionsRequest_v3(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:14.342 7f1b5af81a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 2 (timeout_ms 1600.0): ApiVersionsRequest_v3(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:04:14.342 7f1b5af81a40 [kafka.client] [DEBUG]: Timeouts: user 199.977875, metadata inf, idle connection 539986.000000, request 1599.799156 09/01/2025 02:04:14.344 7f1b5af81a40 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:04:14.344 7f1b5af81a40 [kafka.protocol.parser] [DEBUG]: Processing response ApiVersionsResponse_v3 09/01/2025 02:04:14.344 7f1b5af81a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 2 (2.687692642211914 ms): ApiVersionsResponse_v3(error_code=0, api_versions=[(api_key=0, min_version=0, max_version=8, _tagged_fields={}), (api_key=1, min_version=0, max_version=11, _tagged_fields={}), (api_key=2, min_version=0, max_version=5, _tagged_fields={}), (api_key=3, min_version=0, max_version=9, _tagged_fields={}), (api_key=4, min_version=0, max_version=4, _tagged_fields={}), (api_key=5, min_version=0, max_version=3, _tagged_fields={}), (api_key=6, min_version=0, max_version=6, _tagged_fields={}), (api_key=7, min_version=0, max_version=3, _tagged_fields={}), (api_key=8, min_version=0, max_version=8, _tagged_fields={}), (api_key=9, min_version=0, max_version=7, _tagged_fields={}), (api_key=10, min_version=0, max_version=3, _tagged_fields={}), (api_key=11, min_version=0, max_version=7, _tagged_fields={}), (api_key=12, min_version=0, max_version=4, _tagged_fields={}), (api_key=13, min_version=0, max_version=4, _tagged_fields={}), (api_key=14, min_version=0, max_version=5, _tagged_fields={}), (api_key=15, min_version=0, max_version=5, _tagged_fields={}), (api_key=16, min_version=0, max_version=4, _tagged_fields={}), (api_key=17, min_version=0, max_version=1, _tagged_fields={}), (api_key=18, min_version=0, max_version=3, _tagged_fields={}), (api_key=19, min_version=0, max_version=5, _tagged_fields={}), (api_key=20, min_version=0, max_version=4, _tagged_fields={}), (api_key=21, min_version=0, max_version=2, _tagged_fields={}), (api_key=22, min_version=0, max_version=3, _tagged_fields={}), (api_key=23, min_version=0, max_version=3, _tagged_fields={}), (api_key=24, min_version=0, max_version=1, _tagged_fields={}), (api_key=25, min_version=0, max_version=1, _tagged_fields={}), (api_key=26, min_version=0, max_version=1, _tagged_fields={}), (api_key=27, min_version=0, max_version=0, _tagged_fields={}), (api_key=28, min_version=0, max_version=3, _tagged_fields={}), (api_key=29, min_version=0, max_version=2, _tagged_fields={}), (api_key=30, min_version=0, max_version=2, _tagged_fields={}), (api_key=31, min_version=0, max_version=2, _tagged_fields={}), (api_key=32, min_version=0, max_version=3, _tagged_fields={}), (api_key=33, min_version=0, max_version=1, _tagged_fields={}), (api_key=34, min_version=0, max_version=1, _tagged_fields={}), (api_key=35, min_version=0, max_version=2, _tagged_fields={}), (api_key=36, min_version=0, max_version=2, _tagged_fields={}), (api_key=37, min_version=0, max_version=2, _tagged_fields={}), (api_key=38, min_version=0, max_version=2, _tagged_fields={}), (api_key=39, min_version=0, max_version=2, _tagged_fields={}), (api_key=40, min_version=0, max_version=2, _tagged_fields={}), (api_key=41, min_version=0, max_version=2, _tagged_fields={}), (api_key=42, min_version=0, max_version=2, _tagged_fields={}), (api_key=43, min_version=0, max_version=2, _tagged_fields={}), (api_key=44, min_version=0, max_version=1, _tagged_fields={}), (api_key=45, min_version=0, max_version=0, _tagged_fields={}), (api_key=46, min_version=0, max_version=0, _tagged_fields={}), (api_key=47, min_version=0, max_version=0, _tagged_fields={}), (api_key=48, min_version=0, max_version=0, _tagged_fields={}), (api_key=49, min_version=0, max_version=0, _tagged_fields={})], throttle_time_ms=0, _tagged_fields={}) 09/01/2025 02:04:14.345 7f1b5af81a40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: Broker version identified as 2.6 09/01/2025 02:04:14.345 7f1b5af81a40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: Connection complete. 09/01/2025 02:04:14.345 7f1b5af81a40 [kafka.client] [DEBUG]: Node bootstrap-2 connected 09/01/2025 02:04:14.345 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name batch-size 09/01/2025 02:04:14.346 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name compression-rate 09/01/2025 02:04:14.346 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name queue-time 09/01/2025 02:04:14.346 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name records-per-request 09/01/2025 02:04:14.346 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes 09/01/2025 02:04:14.346 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name record-retries 09/01/2025 02:04:14.346 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name errors 09/01/2025 02:04:14.346 7f1b5af81a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name record-size-max 09/01/2025 02:04:14.346 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Starting Kafka producer I/O thread. 09/01/2025 02:04:14.346 7f1b5ae7b540 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) to node bootstrap-2 09/01/2025 02:04:14.346 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) 09/01/2025 02:04:14.347 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 3 (timeout_ms 30000): MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) 09/01/2025 02:04:14.347 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.576817, metadata 30000.000000, idle connection 539982.000000, request 29999.980211 09/01/2025 02:04:14.347 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Kafka producer started 09/01/2025 02:04:14.347 7f1b5af81a40 [contrail-alarm-gen] [INFO]: Initialized health-check KafkaProducer 09/01/2025 02:04:14.347 7f1b5af81a40 [contrail-alarm-gen] [INFO]: health-check KafkaProducer call returned 09/01/2025 02:04:14.347 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994764, metadata 30000.000000, idle connection 539981.000000, request 29999.570370 09/01/2025 02:04:14.355 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 02:04:14.355 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:04:14.355 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 3 (8.378982543945312 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[]) 09/01/2025 02:04:14.355 7f1b5ae7b540 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 3, topics: 0, groups: 0) 09/01/2025 02:04:14.356 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993095, metadata 299999.832275, idle connection 539973.000000, request inf 09/01/2025 02:04:14.494 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:14.495 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:14.495 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:14.495 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:14.495 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:14.495 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:14.495 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:14.496 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:14.496 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.496 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:14.496 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.496 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.496 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:14.496 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.710 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:15.711 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:16.712 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:17.713 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:18.496 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:18.497 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:18.497 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:18.497 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:18.498 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:18.498 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:18.498 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:18.498 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:18.498 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.498 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:18.498 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.498 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.498 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:18.498 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.715 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:19.716 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:20.718 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:21.718 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:22.499 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:22.500 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:22.500 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:22.500 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:22.500 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:22.500 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:22.500 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:22.501 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:22.501 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.501 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:22.501 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.501 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.501 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:22.501 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.719 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:23.721 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:24.722 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:25.723 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:26.501 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:26.503 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:26.503 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:26.503 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:26.503 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:26.503 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:26.503 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:26.504 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:26.504 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.504 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:26.504 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.504 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.504 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:26.504 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.724 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:27.724 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:28.726 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:29.727 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:30.504 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:30.505 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:30.505 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:30.505 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:30.506 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:30.506 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:30.506 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:30.506 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:30.506 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.506 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:30.506 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.506 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.506 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:30.506 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.729 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:31.730 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:32.732 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:33.733 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:34.506 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:34.507 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:34.507 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:34.507 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:34.507 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:34.507 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:34.507 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:34.508 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:34.508 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.508 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:34.508 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.508 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.508 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:34.508 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.734 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:35.736 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:36.737 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:37.738 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:38.507 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:38.508 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:38.508 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:38.508 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:38.509 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:38.509 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:38.509 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:38.509 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:38.509 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.509 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:38.509 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.509 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.509 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:38.509 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.740 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:39.741 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:40.742 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:41.743 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:42.508 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:42.510 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:42.510 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:42.510 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:42.511 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:42.511 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:42.511 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:42.511 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:42.511 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.511 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:42.511 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.511 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.511 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:42.511 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.744 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:43.746 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:44.747 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:45.749 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:46.511 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:46.512 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:46.512 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:46.512 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:46.512 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:46.512 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:46.512 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:46.513 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:46.513 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.513 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:46.513 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.513 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.513 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:46.513 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.749 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:47.751 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:48.751 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:49.753 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:50.514 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:50.515 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:50.515 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:50.515 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:50.515 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:50.515 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:50.515 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:50.516 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:50.516 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.516 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:50.516 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.516 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.516 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:50.516 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.755 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:51.755 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:52.756 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:53.758 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:54.516 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:54.517 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:54.517 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:54.517 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:54.518 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:54.518 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:54.518 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:54.518 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:54.518 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.518 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:54.518 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.518 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.518 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:54.518 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.759 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:55.760 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:56.761 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:57.762 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:58.518 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:58.519 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:58.519 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:58.520 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:58.520 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:58.520 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:58.520 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:58.520 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:58.520 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.520 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:58.520 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.521 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.521 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:58.521 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.763 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:59.765 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:00.766 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:01.767 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:02.520 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:02.522 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:02.522 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:02.522 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:02.522 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:02.522 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:02.522 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:02.522 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:02.523 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.523 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:02.523 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.523 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.523 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:02.523 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.768 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:03.769 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:04.770 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:05.772 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:06.522 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:06.523 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:06.523 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:06.523 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:06.523 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:06.524 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:06.524 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:06.524 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:06.524 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.524 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:06.524 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.524 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.524 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:06.524 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.774 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:07.775 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:08.776 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:09.777 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:10.525 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:10.526 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:10.526 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:10.526 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:10.526 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:10.526 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:10.526 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:10.526 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:10.526 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.527 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:10.527 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.527 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.527 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:10.527 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.777 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:11.779 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:12.780 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:13.781 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:14.333 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:05:14.334 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.334 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:05:14.334 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.526 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:14.528 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:14.528 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:14.528 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:14.528 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:14.528 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:14.528 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:14.528 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:14.528 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.529 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:14.529 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.529 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.529 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:14.529 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.782 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:15.783 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:16.785 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:17.786 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:18.528 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:18.529 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:18.530 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:18.530 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:18.530 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:18.530 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:18.530 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:18.530 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:18.530 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.530 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:18.530 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.531 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.531 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:18.531 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.788 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:19.789 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:20.791 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:21.792 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:22.530 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:22.532 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:22.532 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:22.532 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:22.532 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:22.532 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:22.532 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:22.533 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:22.533 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.533 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:22.533 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.533 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.533 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:22.533 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.794 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:23.795 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:24.796 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:25.797 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:26.533 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:26.534 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:26.534 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:26.534 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:26.534 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:26.534 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:26.534 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:26.535 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:26.535 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.535 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:26.535 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.535 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.535 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:26.535 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.799 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:27.799 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:28.801 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:29.803 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:30.534 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:30.536 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:30.536 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:30.536 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:30.536 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:30.536 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:30.537 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:30.537 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:30.537 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.537 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:30.537 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.537 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.537 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:30.537 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.806 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:31.807 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:32.808 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:33.810 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:34.537 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:34.538 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:34.538 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:34.538 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:34.539 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:34.539 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:34.539 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:34.539 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:34.539 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.539 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:34.539 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.540 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.540 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:34.540 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.811 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:35.813 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:36.814 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:37.815 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:38.539 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:38.540 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:38.540 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:38.541 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.816 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:39.817 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:40.819 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:41.825 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:42.541 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:42.542 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.543 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:42.544 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.827 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:43.828 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:44.830 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:45.831 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:46.544 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:46.545 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:46.545 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:46.545 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:46.545 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:46.545 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:46.545 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:46.546 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:46.546 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.546 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:46.546 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.546 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.546 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:46.546 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.832 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:47.834 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:48.835 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:49.836 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:50.546 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:50.548 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:50.548 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:50.548 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:50.548 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:50.548 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:50.548 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:50.548 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:50.549 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.549 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:50.549 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.549 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.549 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:50.549 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.838 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:51.840 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:52.841 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:53.842 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:54.548 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:54.549 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:54.550 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:54.550 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:54.550 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:54.550 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:54.550 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:54.550 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:54.550 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.550 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:54.550 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.551 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.551 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:54.551 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.843 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:55.845 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:56.846 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:57.848 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:58.551 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:58.552 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:58.552 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:58.552 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:58.552 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:58.552 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:58.552 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:58.553 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:58.553 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.553 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:58.553 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.553 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.553 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:58.553 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.849 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:59.851 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:00.852 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:01.854 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:02.553 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:02.554 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:02.555 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:02.555 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:02.555 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:02.555 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:02.555 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:02.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:02.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.556 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:02.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.556 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:02.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.856 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:03.858 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:04.859 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:05.861 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:06.555 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:06.556 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:06.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:06.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:06.556 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:06.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:06.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:06.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:06.556 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.557 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:06.557 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.557 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.557 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:06.557 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.862 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:07.864 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:08.865 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:09.866 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:10.557 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:10.559 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:10.559 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:10.559 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:10.559 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:10.559 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:10.560 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:10.560 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:10.560 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.560 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:10.560 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.560 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.561 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:10.561 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.867 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:11.869 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:12.871 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:13.872 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:14.334 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:06:14.335 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.335 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:06:14.335 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.347 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:14.348 7f1b5ae7b540 [kafka.client] [DEBUG]: Initializing connection to node 3 for metadata request 09/01/2025 02:06:14.348 7f1b5ae7b540 [kafka.client] [DEBUG]: Initiating connection to node 3 at 10.0.0.131:9092 09/01/2025 02:06:14.348 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-3.bytes-sent 09/01/2025 02:06:14.348 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-3.bytes-received 09/01/2025 02:06:14.348 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-3.latency 09/01/2025 02:06:14.348 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-3.throttle 09/01/2025 02:06:14.349 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:06:14.349 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:06:14.349 7f1b5ae7b540 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: connecting to 10.0.0.131:9092 [('10.0.0.131', 9092) IPv4] 09/01/2025 02:06:14.349 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999988.759995, metadata inf, idle connection 419979.000000, request inf 09/01/2025 02:06:14.349 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: established TCP connection 09/01/2025 02:06:14.350 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: checking broker Api Versions 09/01/2025 02:06:14.350 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Using pre-configured api_version (2, 6) for ApiVersions 09/01/2025 02:06:14.350 7f1b5ae7b540 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: Connection complete. 09/01/2025 02:06:14.350 7f1b5ae7b540 [kafka.client] [DEBUG]: Node 3 connected 09/01/2025 02:06:14.350 7f1b5ae7b540 [kafka.conn] [INFO]: [IPv4 ('10.0.0.131', 9092)]>: Closing connection. 09/01/2025 02:06:14.350 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: reconnect backoff 0.05533009719565055 after 1 failures 09/01/2025 02:06:14.350 7f1b5ae7b540 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 3 09/01/2025 02:06:14.350 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.350 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 1 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.350 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.635944, metadata 30000.000000, idle connection 419978.000000, request 29999.979258 09/01/2025 02:06:14.351 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994287, metadata 30000.000000, idle connection 419977.000000, request 29999.670744 09/01/2025 02:06:14.405 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:06:14.405 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:14.405 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 1 (54.619789123535156 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:14.405 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992142, metadata 99.966797, idle connection 419923.000000, request inf 09/01/2025 02:06:14.406 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.05822610855102539 secs. 09/01/2025 02:06:14.406 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:14.406 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993334, metadata 99.458252, idle connection 419922.000000, request inf 09/01/2025 02:06:14.506 7f1b5ae7b540 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 3 09/01/2025 02:06:14.507 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.507 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 2 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.507 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.454031, metadata 30000.000000, idle connection 419821.000000, request 29999.973536 09/01/2025 02:06:14.507 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993095, metadata 30000.000000, idle connection 419821.000000, request 29999.604225 09/01/2025 02:06:14.524 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:06:14.525 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:14.525 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 2 (17.82083511352539 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:14.525 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992142, metadata 99.965576, idle connection 419803.000000, request inf 09/01/2025 02:06:14.525 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.17779850959777832 secs. 09/01/2025 02:06:14.525 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:14.525 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994526, metadata 99.545654, idle connection 419803.000000, request inf 09/01/2025 02:06:14.560 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:14.561 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.562 7f1b5ae5d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:14.563 7f1b5ae5d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.626 7f1b5ae7b540 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 3 09/01/2025 02:06:14.626 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.626 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 3 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.626 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.446402, metadata 30000.000000, idle connection 419702.000000, request 29999.962807 09/01/2025 02:06:14.627 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993095, metadata 30000.000000, idle connection 419701.000000, request 29999.535561 09/01/2025 02:06:14.632 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 02:06:14.633 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:14.633 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 3 (6.193876266479492 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:14.633 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991665, metadata 99.967041, idle connection 419695.000000, request inf 09/01/2025 02:06:14.633 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.2857515811920166 secs. 09/01/2025 02:06:14.633 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:14.633 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994049, metadata 99.537354, idle connection 419695.000000, request inf 09/01/2025 02:06:14.734 7f1b5ae7b540 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 3 09/01/2025 02:06:14.734 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.734 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 4 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.734 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.433527, metadata 30000.000000, idle connection 419594.000000, request 29999.969482 09/01/2025 02:06:14.735 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992142, metadata 30000.000000, idle connection 419593.000000, request 29999.470711 09/01/2025 02:06:14.740 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 4 09/01/2025 02:06:14.740 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:14.740 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 4 (5.957841873168945 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:14.741 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990711, metadata 99.963867, idle connection 419588.000000, request inf 09/01/2025 02:06:14.741 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.3934445381164551 secs. 09/01/2025 02:06:14.741 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:14.741 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994287, metadata 99.532959, idle connection 419587.000000, request inf 09/01/2025 02:06:14.842 7f1b5ae7b540 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 3 09/01/2025 02:06:14.842 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.842 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 5 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.842 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.345312, metadata 30000.000000, idle connection 419486.000000, request 29999.973059 09/01/2025 02:06:14.842 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993572, metadata 30000.000000, idle connection 419486.000000, request 29999.622583 09/01/2025 02:06:14.848 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 5 09/01/2025 02:06:14.848 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:14.848 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 5 (6.293058395385742 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:06:14.849 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993095, metadata 99.970215, idle connection 419480.000000, request inf 09/01/2025 02:06:14.849 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.5014941692352295 secs. 09/01/2025 02:06:14.849 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:06:14.849 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993811, metadata 99.500977, idle connection 419479.000000, request inf 09/01/2025 02:06:14.873 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:14.949 7f1b5ae7b540 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 3 09/01/2025 02:06:14.949 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.950 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 6 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:06:14.950 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.464521, metadata 30000.000000, idle connection 419379.000000, request 29999.974251 09/01/2025 02:06:14.950 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993334, metadata 30000.000000, idle connection 419378.000000, request 29999.634027 09/01/2025 02:06:14.958 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 6 09/01/2025 02:06:14.958 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:06:14.958 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 6 (8.782386779785156 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=5, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=10, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=20, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=25, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=26, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=11, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=4, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=16, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=17, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=23, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=28, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=13, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=22, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=8, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=2, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=29, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=19, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=14, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=1, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=7, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[])])]) 09/01/2025 02:06:14.959 7f1b5ae7b540 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 3, topics: 1, groups: 0) 09/01/2025 02:06:14.960 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990711, metadata 299999.880859, idle connection 419369.000000, request inf 09/01/2025 02:06:14.960 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.6123185157775879 secs. 09/01/2025 02:06:14.960 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=29) 09/01/2025 02:06:14.960 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=29) is either full or getting a new batch 09/01/2025 02:06:14.960 7f1b5af81a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:06:14.960 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:06:14.960 7f1b5af81a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=29) 09/01/2025 02:06:14.960 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.records-per-batch 09/01/2025 02:06:14.961 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.bytes 09/01/2025 02:06:14.961 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.compression-rate 09/01/2025 02:06:14.961 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.record-retries 09/01/2025 02:06:14.961 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.record-errors 09/01/2025 02:06:14.961 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {3} 09/01/2025 02:06:14.961 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {3: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=29, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x07\x18u\xc7\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06e\xb0\x00\x00\x01\x99\x03\x06e\xb0\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:06:14.961 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=29, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x07\x18u\xc7\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06e\xb0\x00\x00\x01\x99\x03\x06e\xb0\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:06:14.961 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=29, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x07\x18u\xc7\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06e\xb0\x00\x00\x01\x99\x03\x06e\xb0\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:06:14.961 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 7 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=29, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x07\x18u\xc7\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x06e\xb0\x00\x00\x01\x99\x03\x06e\xb0\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:06:14.962 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 299997.727295, idle connection 419367.000000, request 29999.965429 09/01/2025 02:06:14.962 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993811, metadata 299997.317139, idle connection 419366.000000, request 29999.559879 09/01/2025 02:06:15.029 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 7 09/01/2025 02:06:15.029 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:06:15.029 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 7 (67.25096702575684 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=29, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:06:15.029 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=29, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:06:15.029 7f1b5ae7b540 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=29) with base offset 0 log start offset 0. 09/01/2025 02:06:15.030 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.989758, metadata 299929.803711, idle connection 419299.000000, request inf 09/01/2025 02:06:15.030 7f1b5af81a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:06:15.874 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:16.876 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:17.878 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:18.562 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:18.563 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:18.564 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:18.564 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:18.564 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:18.564 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:18.564 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:18.564 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:18.564 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.564 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:18.564 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.564 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.565 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:18.565 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.879 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:19.880 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:20.883 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:21.884 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:22.565 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:22.566 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:22.566 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:22.566 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:22.566 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:22.566 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:22.566 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:22.566 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:22.566 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.567 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:22.567 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.567 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.567 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:22.567 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.885 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:23.887 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:24.889 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:25.889 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:26.567 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:26.568 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:26.568 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:26.568 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:26.568 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:26.568 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:26.568 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:26.569 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:26.569 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.569 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:26.569 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.569 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.569 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:26.569 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.891 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:27.892 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:28.893 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:29.895 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:30.568 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:30.569 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:30.569 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:30.570 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:30.570 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:30.570 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:30.570 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:30.570 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:30.570 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.570 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:30.570 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.570 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.570 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:30.571 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.896 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:31.898 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:32.899 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:33.900 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:34.570 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:34.571 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:34.571 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:34.571 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:34.571 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:34.571 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:34.571 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:34.571 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:34.571 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.571 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:34.572 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.572 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.572 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:34.572 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.901 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:35.903 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:36.904 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:37.905 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:38.572 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:38.573 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:38.573 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:38.573 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:38.573 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:38.573 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:38.573 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:38.574 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:38.574 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.574 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:38.574 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.574 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.574 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:38.574 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.906 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:39.907 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:40.908 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:41.909 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:42.574 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:42.575 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:42.575 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:42.575 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.575 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:42.575 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.575 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:42.575 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:42.576 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.576 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:42.576 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.576 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.576 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:42.576 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.910 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:43.912 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:44.913 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:45.915 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:46.576 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:46.577 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:46.578 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:46.578 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.578 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:46.578 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.578 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:46.578 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:46.578 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.579 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:46.579 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.579 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.579 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:46.579 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.916 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:47.918 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:48.919 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:49.921 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:50.578 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:50.580 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:50.580 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:50.580 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.580 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:50.580 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.580 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:50.580 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:50.581 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.581 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:50.581 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.581 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.581 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:50.581 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.923 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:51.924 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:52.926 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:53.928 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:54.580 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:54.581 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:54.581 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:54.582 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.582 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:54.582 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.582 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:54.582 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:54.582 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.582 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:54.582 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.582 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.582 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:54.583 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.928 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:55.929 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:56.931 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:57.932 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:58.583 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:58.584 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:58.584 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:58.584 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.584 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:58.584 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.585 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:58.585 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:58.585 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.585 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:58.585 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.585 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.585 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:58.585 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.934 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:59.936 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:00.937 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:01.939 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:02.586 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:02.587 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:02.587 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:02.587 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.587 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:02.587 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.588 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:02.588 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:02.588 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.588 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:02.588 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.588 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.588 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:02.588 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.940 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:03.942 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:04.944 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:05.944 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:06.588 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:06.589 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:06.589 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:06.589 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.590 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:06.590 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.590 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:06.590 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:06.590 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.590 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:06.590 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.590 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.590 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:06.590 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.946 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:07.948 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:08.950 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:09.952 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:10.589 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:10.590 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:10.590 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:10.591 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.953 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:11.954 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:12.956 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:13.957 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:14.335 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:07:14.336 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.336 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:07:14.336 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.591 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:14.592 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:14.592 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:14.592 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.593 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:14.593 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.593 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:14.593 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:14.593 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.593 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:14.593 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.593 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.593 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:14.593 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.959 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:15.961 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:16.962 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:17.964 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:18.593 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:18.594 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:18.595 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:18.595 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.595 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:18.595 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.595 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:18.595 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:18.595 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.595 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:18.595 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.595 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.596 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:18.596 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.965 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:19.967 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:20.969 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:21.970 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:22.595 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:22.596 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:22.596 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:22.597 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.971 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:23.972 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:24.973 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:25.975 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:26.597 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:26.599 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:26.599 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:26.599 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:26.599 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:26.599 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:26.599 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:26.599 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:26.599 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.600 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:26.600 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.600 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.600 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:26.600 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.976 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:27.978 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:28.980 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:29.981 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:30.599 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:30.600 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:30.600 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:30.601 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:30.601 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:30.601 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:30.601 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:30.601 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:30.601 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.601 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:30.601 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.601 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.601 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:30.602 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.983 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:31.985 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:32.986 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:33.988 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:34.601 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:34.602 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:34.602 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:34.602 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:34.602 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:34.603 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:34.603 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:34.603 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:34.603 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.603 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:34.603 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.603 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.603 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:34.603 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.989 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:35.991 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:36.993 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:37.994 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:38.604 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:38.605 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:38.605 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:38.605 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:38.605 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:38.605 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:38.605 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:38.606 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:38.606 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.606 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:38.606 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.606 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.606 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:38.606 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.996 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:39.997 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:40.998 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:42.000 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:42.606 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:42.607 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:42.607 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:42.607 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:42.608 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:42.608 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:42.608 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:42.608 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:42.608 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.608 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:42.608 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.608 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.608 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:42.608 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.002 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:44.004 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:45.005 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:46.007 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:46.609 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:46.610 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:46.610 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:46.610 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:46.610 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:46.610 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:46.610 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:46.611 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:46.611 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.611 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:46.611 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.611 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.611 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:46.611 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.008 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:48.010 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:49.012 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:50.014 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:50.611 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:50.612 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:50.613 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:50.613 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:50.613 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:50.613 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:50.613 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:50.613 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:50.613 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.613 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:50.614 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.614 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.614 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:50.614 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.015 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:52.017 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:53.019 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:54.020 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:54.613 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:54.614 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:54.615 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:54.615 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:54.615 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:54.615 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:54.615 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:54.615 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:54.615 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.615 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:54.615 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.615 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.616 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:54.616 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.021 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:56.023 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:57.024 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:58.025 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:58.616 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:58.617 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:58.617 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:58.617 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:58.618 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:58.618 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:58.618 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:58.618 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:58.618 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.618 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:58.618 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.618 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.618 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:58.618 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.027 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:00.028 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:01.030 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:02.031 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:02.618 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:02.619 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:02.619 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:02.620 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:02.620 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:02.620 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:02.620 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:02.620 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:02.620 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.620 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:02.620 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.620 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.621 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:02.621 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.032 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:04.034 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:05.036 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:06.037 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:06.620 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:06.621 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:06.621 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:06.621 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:06.622 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:06.622 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:06.622 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:06.622 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:06.622 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.622 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:06.622 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.622 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.622 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:06.622 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.039 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:08.041 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:09.042 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:10.044 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:10.622 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:10.623 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:10.624 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:10.624 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:10.624 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:10.624 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:10.624 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:10.625 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:10.625 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.625 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:10.625 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.625 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.625 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:10.625 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.045 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:12.047 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:13.049 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:14.050 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:14.336 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:08:14.337 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.337 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:08:14.337 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.624 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:14.626 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:14.626 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:14.626 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:14.626 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:14.626 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:14.626 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:14.627 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:14.627 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.627 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:14.627 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.627 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.627 7f1b5aebe4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:14.627 7f1b5aebe4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.031 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=25) 09/01/2025 02:08:15.031 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=25) is either full or getting a new batch 09/01/2025 02:08:15.031 7f1b5af81a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:08:15.031 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:08:15.031 7f1b5af81a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=25) 09/01/2025 02:08:15.032 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Node 2 not ready; delaying produce of accumulated batch (0.000000 ms) 09/01/2025 02:08:15.032 7f1b5ae7b540 [kafka.client] [DEBUG]: Initiating connection to node 2 at 10.0.0.38:9092 09/01/2025 02:08:15.032 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-2.bytes-sent 09/01/2025 02:08:15.032 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-2.bytes-received 09/01/2025 02:08:15.032 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-2.latency 09/01/2025 02:08:15.032 7f1b5ae7b540 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-2.throttle 09/01/2025 02:08:15.033 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:08:15.033 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:08:15.033 7f1b5ae7b540 [kafka.conn] [INFO]: [IPv4 ('10.0.0.38', 9092)]>: connecting to 10.0.0.38:9092 [('10.0.0.38', 9092) IPv4] 09/01/2025 02:08:15.033 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 179926.312256, idle connection 299295.000000, request inf 09/01/2025 02:08:15.033 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Node 2 not ready; delaying produce of accumulated batch (49.118805 ms) 09/01/2025 02:08:15.033 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 49.108744, metadata 179925.958984, idle connection 299295.000000, request inf 09/01/2025 02:08:15.034 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: established TCP connection 09/01/2025 02:08:15.034 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: checking broker Api Versions 09/01/2025 02:08:15.034 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: Using pre-configured api_version (2, 6) for ApiVersions 09/01/2025 02:08:15.034 7f1b5ae7b540 [kafka.conn] [INFO]: [IPv4 ('10.0.0.38', 9092)]>: Connection complete. 09/01/2025 02:08:15.034 7f1b5ae7b540 [kafka.client] [DEBUG]: Node 2 connected 09/01/2025 02:08:15.034 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {2} 09/01/2025 02:08:15.034 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {2: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=25, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02bQ\xaa*\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08:\xb7\x00\x00\x01\x99\x03\x08:\xb7\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:08:15.034 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=25, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02bQ\xaa*\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08:\xb7\x00\x00\x01\x99\x03\x08:\xb7\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:08:15.035 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=25, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02bQ\xaa*\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08:\xb7\x00\x00\x01\x99\x03\x08:\xb7\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:08:15.035 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: Request 1 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=25, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02bQ\xaa*\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x08:\xb7\x00\x00\x01\x99\x03\x08:\xb7\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:08:15.035 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 179924.433838, idle connection 299293.000000, request 29999.965429 09/01/2025 02:08:15.035 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.994287, metadata 179924.024170, idle connection 299293.000000, request 29999.560595 09/01/2025 02:08:15.039 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:08:15.039 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:08:15.039 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: Response 1 (4.357576370239258 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=25, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:08:15.039 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=25, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:08:15.040 7f1b5ae7b540 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=25) with base offset 0 log start offset 0. 09/01/2025 02:08:15.040 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992857, metadata 179919.709717, idle connection 299289.000000, request inf 09/01/2025 02:08:15.040 7f1b5af81a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:08:15.051 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:16.053 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:17.054 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:18.056 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:18.626 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:18.628 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:18.628 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:18.628 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:18.628 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:18.628 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:18.628 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:18.628 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:18.629 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.629 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:18.629 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.629 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.629 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:18.629 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.057 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:20.059 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:21.059 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:22.061 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:22.629 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:22.630 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:22.630 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:22.631 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:22.631 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:22.631 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:22.631 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:22.631 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:22.631 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.631 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:22.631 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.631 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.631 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:22.632 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.062 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:24.063 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:25.065 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:26.066 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:26.631 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:26.633 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:26.633 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:26.633 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:26.633 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:26.633 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:26.633 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:26.633 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:26.634 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.634 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:26.634 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.634 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.634 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:26.634 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.067 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:28.068 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:29.070 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:30.072 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:30.633 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:30.634 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:30.634 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:30.634 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:30.634 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:30.635 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:30.635 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:30.635 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:30.635 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.635 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:30.635 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.635 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.635 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:30.635 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.073 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:32.074 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:33.076 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:34.078 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:34.635 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:34.636 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:34.636 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:34.636 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:34.636 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:34.636 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:34.637 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:34.637 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:34.637 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.637 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:34.637 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.637 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.637 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:34.637 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.079 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:36.080 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:37.082 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:38.084 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:38.637 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:38.638 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:38.638 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:38.638 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:38.639 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:38.639 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:38.639 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:38.639 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:38.639 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.639 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:38.639 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.639 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.639 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:38.640 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.085 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:40.086 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:41.087 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:42.088 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:42.639 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:42.640 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:42.640 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:42.640 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:42.641 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:42.641 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:42.641 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:42.641 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:42.641 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.641 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:42.641 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.641 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.641 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:42.641 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.090 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:44.092 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:45.093 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:46.095 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:46.641 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:46.642 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:46.642 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:46.642 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:46.642 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:46.642 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:46.642 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:46.643 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:46.643 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.643 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:46.643 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.643 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.643 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:46.643 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.096 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:48.097 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:49.099 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:50.101 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:50.643 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:50.644 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:50.644 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:50.644 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:50.645 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:50.645 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:50.645 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:50.645 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:50.645 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.645 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:50.645 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.645 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.645 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:50.645 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.102 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:52.103 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:53.105 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:54.106 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:54.644 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:54.645 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:54.645 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:54.645 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:54.646 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:54.646 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:54.646 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:54.646 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:54.646 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.646 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:54.646 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.646 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.646 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:54.646 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.108 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:56.110 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:57.111 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:58.113 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:58.646 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:58.647 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:58.648 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:58.648 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:58.648 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:58.648 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:58.648 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:58.648 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:58.648 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.648 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:58.648 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.648 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.649 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:58.649 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.114 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:00.116 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:01.117 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:02.119 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:02.648 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:02.650 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:02.650 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:02.650 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:02.650 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:02.650 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:02.650 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:02.650 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:02.651 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.651 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:02.651 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.651 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.651 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:02.651 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.121 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:04.123 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:05.124 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:06.126 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:06.651 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:06.652 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:06.652 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:06.652 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:06.652 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:06.652 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:06.652 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:06.653 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:06.653 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.653 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:06.653 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.653 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.653 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:06.653 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.127 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:08.128 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:09.130 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:10.131 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:10.653 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:10.654 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:10.654 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:10.654 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:10.654 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:10.654 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:10.654 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:10.655 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:10.655 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.655 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:10.655 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.655 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.655 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:10.655 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.132 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:12.133 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:13.135 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:14.136 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:14.337 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:09:14.337 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.337 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:09:14.337 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.654 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:14.655 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.656 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:14.657 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.137 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:16.139 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:17.140 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:18.141 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:18.656 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:18.657 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:18.658 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.142 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:20.144 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:21.145 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:22.146 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:22.659 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:22.659 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:22.660 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:22.660 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:22.660 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:22.660 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:22.660 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:22.660 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:22.660 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.660 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:22.660 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.660 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.661 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:22.661 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.148 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:24.150 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:25.152 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:26.154 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:26.660 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:26.662 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:26.662 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:26.662 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:26.662 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:26.662 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:26.662 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:26.662 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:26.662 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.663 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:26.663 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.663 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.663 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:26.663 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.155 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:28.157 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:29.158 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:30.159 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:30.662 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:30.663 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:30.664 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:30.664 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:30.664 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:30.664 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:30.664 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:30.664 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:30.664 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.664 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:30.664 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.664 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.665 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:30.665 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.160 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:32.162 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:33.164 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:34.166 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:34.664 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:34.665 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:34.665 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:34.666 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.167 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:36.168 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:37.170 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:38.171 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:38.667 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:38.668 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:38.668 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:38.668 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:38.669 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:38.669 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:38.669 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:38.669 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:38.669 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.669 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:38.669 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.669 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.669 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:38.669 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.172 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:40.174 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:41.175 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:42.176 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:42.670 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:42.671 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:42.671 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:42.672 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:42.672 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:42.672 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:42.672 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:42.672 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:42.672 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.672 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:42.673 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.673 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.673 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:42.673 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.177 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:44.179 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:45.181 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:46.182 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:46.672 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:46.707 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:46.707 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:46.708 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:46.708 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:46.708 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:46.708 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:46.708 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:46.708 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.708 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:46.708 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.709 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.709 7f1b5aebe5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:46.709 7f1b5aebe5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.184 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:48.185 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:49.187 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:50.188 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:50.709 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:50.710 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:50.710 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:50.710 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:50.710 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:50.710 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:50.711 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:50.711 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:50.711 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.711 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:50.711 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.711 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.711 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:50.711 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.190 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:52.192 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:53.193 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:54.194 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:54.711 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:54.712 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:54.712 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:54.712 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:54.713 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:54.713 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:54.713 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:54.713 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:54.713 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.713 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:54.713 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.713 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.714 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:54.714 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.196 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:56.197 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:57.199 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:58.200 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:58.713 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:58.714 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:58.714 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:58.714 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:58.714 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:58.715 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:58.715 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:58.715 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:58.715 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.715 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:58.715 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.715 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.715 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:58.715 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.202 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:00.204 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:01.205 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:02.207 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:02.716 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:02.717 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:02.717 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:02.717 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:02.717 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:02.717 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:02.718 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:02.718 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:02.718 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.718 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:02.718 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.718 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.718 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:02.718 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.208 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:04.210 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:05.212 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:06.213 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:06.718 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:06.719 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:06.719 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:06.719 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.719 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:06.720 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.720 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:06.720 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:06.720 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.720 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:06.720 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.720 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.720 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:06.720 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.215 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:08.217 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:09.218 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:10.220 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:10.720 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:10.721 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:10.721 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:10.721 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.721 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:10.721 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.721 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:10.721 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:10.721 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.722 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:10.722 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.722 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.722 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:10.722 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.221 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:12.223 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:13.225 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:14.227 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:14.337 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:10:14.338 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.338 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:10:14.338 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.722 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:14.723 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:14.723 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:14.723 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.723 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:14.723 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.723 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:14.724 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:14.724 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.724 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:14.724 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.724 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.725 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:14.725 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.041 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=20) 09/01/2025 02:10:15.042 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=20) is either full or getting a new batch 09/01/2025 02:10:15.042 7f1b5af81a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:10:15.042 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:10:15.042 7f1b5af81a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=20) 09/01/2025 02:10:15.043 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {3} 09/01/2025 02:10:15.043 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {3: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=20, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02=45\x8a\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n\x0f\x82\x00\x00\x01\x99\x03\n\x0f\x82\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:10:15.043 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=20, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02=45\x8a\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n\x0f\x82\x00\x00\x01\x99\x03\n\x0f\x82\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:10:15.043 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=20, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02=45\x8a\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n\x0f\x82\x00\x00\x01\x99\x03\n\x0f\x82\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:10:15.043 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 8 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=20, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02=45\x8a\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\n\x0f\x82\x00\x00\x01\x99\x03\n\x0f\x82\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:10:15.043 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 59916.229492, idle connection 179285.000000, request 29999.954462 09/01/2025 02:10:15.044 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993572, metadata 59915.841553, idle connection 179285.000000, request 29999.585152 09/01/2025 02:10:15.048 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 8 09/01/2025 02:10:15.048 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:10:15.048 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 8 (5.263566970825195 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=20, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:10:15.049 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=20, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:10:15.049 7f1b5ae7b540 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=20) with base offset 0 log start offset 0. 09/01/2025 02:10:15.049 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991426, metadata 59910.491943, idle connection 179279.000000, request inf 09/01/2025 02:10:15.049 7f1b5af81a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:10:15.227 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:16.229 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:17.231 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:18.232 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:18.723 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:18.725 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:18.725 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:18.725 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.725 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:18.725 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.725 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:18.725 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:18.726 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.726 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:18.726 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.726 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.726 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:18.726 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.233 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:20.235 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:21.236 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:22.238 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:22.726 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:22.727 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:22.727 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:22.727 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.728 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:22.728 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.728 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:22.728 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:22.728 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.728 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:22.728 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.728 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.728 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:22.728 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.239 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:24.241 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:25.242 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:26.244 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:26.728 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:26.729 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:26.730 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:26.730 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.730 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:26.730 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.730 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:26.730 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:26.730 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.731 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:26.731 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.731 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.731 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:26.731 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.246 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:28.247 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:29.249 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:30.251 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:30.730 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:30.731 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:30.732 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:30.732 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:30.732 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:30.732 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:30.732 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:30.732 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:30.732 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.732 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:30.732 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.733 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.733 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:30.733 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.252 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:32.254 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:33.255 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:34.257 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:34.733 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:34.735 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:34.735 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:34.735 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:34.736 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:34.736 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:34.736 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:34.736 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:34.736 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.736 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:34.736 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.736 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.736 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:34.736 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.259 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:36.260 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:37.262 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:38.263 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:38.736 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:38.738 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:38.738 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:38.738 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:38.738 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:38.738 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:38.738 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:38.738 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:38.738 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.739 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:38.739 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.739 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.739 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:38.739 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.264 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:40.266 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:41.268 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:42.269 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:42.738 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:42.739 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:42.739 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:42.739 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:42.739 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:42.740 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:42.740 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:42.740 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:42.740 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.740 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:42.740 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.740 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.740 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:42.740 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.271 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:44.272 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:45.274 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:46.276 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:46.740 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:46.741 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:46.741 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:46.741 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:46.742 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:46.742 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:46.742 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:46.742 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:46.742 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.742 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:46.742 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.742 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.742 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:46.742 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.278 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:48.279 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:49.281 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:50.283 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:50.742 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:50.743 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:50.743 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:50.743 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:50.744 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:50.744 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:50.744 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:50.744 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:50.744 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.744 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:50.744 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.744 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.744 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:50.744 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.284 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:52.286 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:53.287 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:54.288 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:54.744 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:54.745 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:54.745 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:54.745 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:54.746 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:54.746 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:54.746 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:54.746 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:54.746 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.746 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:54.746 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.746 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.746 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:54.747 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.290 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:56.292 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:57.293 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:58.295 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:58.746 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:58.747 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:58.747 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:58.748 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.297 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:00.298 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:01.301 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:02.302 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:02.748 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:02.749 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:02.749 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:02.749 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:02.750 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:02.750 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:02.750 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:02.750 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:02.750 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.750 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:02.750 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.750 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.750 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:02.750 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.304 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:04.305 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:05.307 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:06.309 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:06.750 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:06.751 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:06.751 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:06.752 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:06.752 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:06.752 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:06.752 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:06.752 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:06.752 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.752 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:06.753 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.753 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.753 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:06.753 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.311 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:08.312 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:09.314 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:10.315 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:10.752 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:10.753 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:10.754 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:10.754 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:10.754 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:10.754 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:10.754 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:10.754 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:10.754 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.755 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:10.755 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.755 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.755 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:10.755 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.317 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:12.318 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:13.320 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:14.322 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:14.338 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:11:14.339 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.339 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:11:14.339 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.755 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:14.756 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:14.756 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:14.756 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:14.756 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:14.756 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:14.756 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:14.757 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:14.757 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.757 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:14.757 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.757 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.757 7f1b5af162c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:14.757 7f1b5af162c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.961 7f1b5ae7b540 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 2 09/01/2025 02:11:14.961 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:11:14.961 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: Request 2 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:11:14.961 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.448309, metadata 30000.000000, idle connection 119367.000000, request 29999.971867 09/01/2025 02:11:14.962 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992142, metadata 30000.000000, idle connection 119367.000000, request 29999.492884 09/01/2025 02:11:14.973 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:11:14.973 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:11:14.974 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.38', 9092)]>: Response 2 (13.083934783935547 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=2, host='10.0.0.38', port=9092, rack=None), (node_id=3, host='10.0.0.131', port=9092, rack=None), (node_id=1, host='10.0.0.249', port=9092, rack=None)], cluster_id='M3e10Rd4TmO0bzHzgqqBwQ', controller_id=2, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=5, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=10, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=20, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=25, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=26, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=11, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=4, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=16, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=17, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=23, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=28, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=13, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=22, leader=2, leader_epoch=0, replicas=[2, 1, 3], isr=[2, 1, 3], offline_replicas=[]), (error_code=0, partition=8, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=2, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=29, leader=3, leader_epoch=0, replicas=[3, 2, 1], isr=[3, 2, 1], offline_replicas=[]), (error_code=0, partition=19, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=14, leader=3, leader_epoch=0, replicas=[3, 1, 2], isr=[3, 1, 2], offline_replicas=[]), (error_code=0, partition=1, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1, 2, 3], isr=[1, 2, 3], offline_replicas=[]), (error_code=0, partition=7, leader=2, leader_epoch=0, replicas=[2, 3, 1], isr=[2, 3, 1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1, 3, 2], isr=[1, 3, 2], offline_replicas=[])])]) 09/01/2025 02:11:14.975 7f1b5ae7b540 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 3, topics: 1, groups: 0) 09/01/2025 02:11:14.976 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991903, metadata 299999.869141, idle connection 119353.000000, request inf 09/01/2025 02:11:15.323 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:16.325 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:17.327 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:18.329 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:18.757 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:18.758 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:18.758 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:18.758 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:18.758 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:18.759 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:18.759 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:18.759 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:18.759 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.759 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:18.759 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.759 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.759 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:18.759 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.330 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:20.332 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:21.333 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:22.335 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:22.759 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:22.760 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:22.760 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:22.761 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:22.761 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:22.761 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:22.761 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:22.761 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:22.761 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.761 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:22.761 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.761 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.762 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:22.762 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.337 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:24.339 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:25.341 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:26.342 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:26.762 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:26.763 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:26.763 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:26.763 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:26.763 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:26.763 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:26.764 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:26.764 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:26.764 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.764 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:26.764 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.764 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.764 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:26.765 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.343 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:28.345 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:29.346 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:30.348 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:30.763 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:30.765 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:30.765 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:30.765 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:30.765 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:30.765 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:30.765 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:30.766 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:30.766 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.766 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:30.766 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.766 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.766 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:30.766 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.349 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:32.351 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:33.353 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:34.355 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:34.766 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:34.767 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:34.767 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:34.767 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:34.767 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:34.767 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:34.767 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:34.767 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:34.768 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.768 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:34.768 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.768 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.768 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:34.768 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.356 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:36.357 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:37.359 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:38.360 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:38.768 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:38.769 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:38.769 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:38.769 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:38.769 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:38.769 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:38.769 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:38.770 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:38.770 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.770 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:38.770 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.770 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.770 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:38.770 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.362 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:40.364 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:41.364 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:42.366 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:42.769 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:42.771 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:42.771 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:42.771 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:42.771 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:42.771 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:42.771 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:42.771 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:42.771 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.771 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:42.772 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.772 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.772 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:42.772 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.367 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:44.369 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:45.370 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:46.372 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:46.771 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:46.773 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:46.773 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:46.773 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:46.773 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:46.773 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:46.773 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:46.774 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:46.774 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.774 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:46.774 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.774 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.774 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:46.774 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.374 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:48.375 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:49.377 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:50.378 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:50.773 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:50.775 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:50.775 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:50.775 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:50.775 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:50.775 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:50.775 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:50.775 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:50.776 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.776 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:50.776 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.776 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.776 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:50.776 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.380 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:52.382 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:53.383 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:54.384 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:54.776 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:54.777 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:54.777 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:54.777 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:54.778 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:54.778 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:54.778 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:54.778 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:54.778 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.778 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:54.778 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.778 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.778 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:54.778 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.385 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:56.387 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:57.388 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:58.390 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:58.778 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:58.779 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:58.779 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:58.780 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:58.780 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:58.780 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:58.780 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:58.780 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:58.780 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.780 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:58.780 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.780 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.781 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:58.781 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.391 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:00.392 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:01.394 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:02.396 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:02.780 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:02.782 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:02.782 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:02.782 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:02.782 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:02.782 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:02.782 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:02.783 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:02.783 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.783 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:02.783 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.783 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.783 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:02.783 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.398 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:04.399 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:05.401 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:06.402 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:06.782 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:06.784 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:06.784 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:06.784 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:06.784 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:06.784 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:06.784 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:06.785 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:06.785 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.785 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:06.785 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.785 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.785 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:06.785 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.404 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:08.405 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:09.406 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:10.407 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:10.785 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:10.786 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:10.786 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:10.786 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:10.786 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:10.787 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:10.787 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:10.787 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:10.787 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.787 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:10.787 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.787 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.787 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:10.787 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.408 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:12.410 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:13.411 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:14.339 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:12:14.339 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.339 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:12:14.339 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.412 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:14.787 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:14.788 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:14.788 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:14.788 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:14.788 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:14.788 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:14.788 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:14.789 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:14.789 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.789 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:14.789 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.789 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.789 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:14.789 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.050 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=11) 09/01/2025 02:12:15.050 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=11) is either full or getting a new batch 09/01/2025 02:12:15.050 7f1b5af81a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:12:15.050 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:12:15.050 7f1b5af81a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=11) 09/01/2025 02:12:15.051 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {3} 09/01/2025 02:12:15.051 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {3: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa3]\xba\xae\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\xe4J\x00\x00\x01\x99\x03\x0b\xe4J\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:12:15.051 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa3]\xba\xae\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\xe4J\x00\x00\x01\x99\x03\x0b\xe4J\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:12:15.052 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa3]\xba\xae\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\xe4J\x00\x00\x01\x99\x03\x0b\xe4J\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:12:15.052 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 9 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa3]\xba\xae\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\xe4J\x00\x00\x01\x99\x03\x0b\xe4J\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:12:15.052 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 239923.557617, idle connection 59276.000000, request 29999.956131 09/01/2025 02:12:15.052 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991188, metadata 239923.085693, idle connection 59276.000000, request 29999.488592 09/01/2025 02:12:15.055 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 9 09/01/2025 02:12:15.055 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:12:15.055 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 9 (3.02886962890625 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:12:15.055 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=11, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:12:15.055 7f1b5ae7b540 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=11) with base offset 0 log start offset 0. 09/01/2025 02:12:15.055 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993095, metadata 239920.113525, idle connection 59273.000000, request inf 09/01/2025 02:12:15.055 7f1b5af81a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:12:15.413 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:16.415 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:17.416 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:18.418 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:18.788 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:18.790 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:18.790 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:18.790 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:18.790 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:18.790 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:18.790 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:18.791 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:18.791 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.791 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:18.791 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.791 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.791 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:18.791 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.420 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:20.422 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:21.423 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:22.425 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:22.791 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:22.792 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:22.792 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:22.792 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:22.793 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:22.793 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:22.793 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:22.793 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:22.793 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.793 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:22.793 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.793 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.793 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:22.793 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.426 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:24.428 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:25.430 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:26.432 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:26.793 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:26.794 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:26.794 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:26.794 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:26.794 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:26.795 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:26.795 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:26.795 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:26.795 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.795 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:26.795 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.795 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.795 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:26.795 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.433 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:28.435 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:29.436 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:30.438 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:30.795 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:30.796 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.797 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:30.798 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.439 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:32.440 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:33.442 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:34.443 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:34.797 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:34.799 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:34.799 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:34.799 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:34.799 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:34.799 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:34.799 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:34.799 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:34.799 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.800 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:34.800 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.800 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.800 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:34.800 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.445 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:36.446 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:37.447 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:38.449 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:38.799 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:38.800 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:38.800 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:38.800 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:38.800 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:38.801 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:38.801 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:38.801 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:38.801 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.801 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:38.801 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.801 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.801 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:38.801 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.451 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:40.453 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:41.455 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:42.456 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:42.801 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:42.802 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:42.802 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:42.802 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:42.802 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:42.802 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:42.802 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:42.803 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:42.803 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.803 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:42.803 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.803 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.803 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:42.803 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.458 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:44.460 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:45.462 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:46.464 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:46.802 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:46.803 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:46.804 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:46.804 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:46.804 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:46.804 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:46.804 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:46.804 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:46.804 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.804 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:46.804 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.804 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.805 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:46.805 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.465 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:48.466 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:49.468 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:50.470 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:50.804 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:50.806 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:50.806 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:50.806 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:50.806 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:50.806 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:50.806 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:50.806 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:50.806 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.806 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:50.806 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.807 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.807 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:50.807 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.472 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:52.473 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:53.475 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:54.476 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:54.807 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:54.808 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:54.808 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:54.808 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:54.808 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:54.809 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:54.809 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:54.809 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:54.809 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.809 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:54.809 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.809 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.809 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:54.809 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.478 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:56.480 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:57.480 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:58.482 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:58.809 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:58.810 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:58.810 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:58.810 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:58.810 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:58.810 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:58.811 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:58.811 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:58.811 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.811 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:58.811 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.811 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.811 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:58.811 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.484 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:00.486 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:01.487 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:02.487 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:02.811 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:02.813 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:02.813 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:02.813 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:02.814 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:02.814 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:02.814 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:02.814 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:02.814 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.815 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:02.815 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.815 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.815 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:02.815 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.489 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:04.489 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:05.491 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:06.493 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:06.814 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:06.816 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:06.816 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:06.816 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:06.816 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:06.816 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:06.816 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:06.817 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:06.817 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.817 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:06.817 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.817 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.817 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:06.817 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.494 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:08.495 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:09.496 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:10.498 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:10.816 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:10.817 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:10.817 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:10.817 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:10.818 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:10.818 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:10.818 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:10.818 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:10.818 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.818 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:10.818 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.818 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.818 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:10.818 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.500 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:12.502 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:13.504 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:14.330 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.990473, metadata 180645.870117, idle connection 420643.000000, request inf 09/01/2025 02:13:14.339 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:13:14.339 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.340 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:13:14.340 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.504 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:14.818 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:14.819 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:14.819 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:14.819 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:14.820 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:14.820 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:14.820 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:14.820 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:14.820 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.820 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:14.820 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.820 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.820 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:14.821 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.506 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:16.508 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:17.509 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:18.511 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:18.820 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:18.821 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:18.822 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:18.822 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:18.822 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:18.822 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:18.822 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:18.822 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:18.823 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.823 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:18.823 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.823 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.823 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:18.823 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.512 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:20.514 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:21.515 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:22.517 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:22.822 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:22.824 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:22.824 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:22.824 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:22.825 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:22.825 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:22.825 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:22.825 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:22.825 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.825 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:22.826 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.826 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.826 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:22.826 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.519 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:24.519 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:25.521 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:26.523 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:26.825 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:26.826 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:26.826 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:26.826 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:26.826 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:26.826 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:26.826 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:26.827 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:26.828 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.828 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:26.828 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.828 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.828 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:26.828 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.525 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:28.527 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:29.528 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:30.529 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:30.827 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:30.828 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:30.828 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:30.828 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:30.828 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:30.828 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:30.828 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:30.829 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:30.829 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.829 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:30.829 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.829 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.829 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:30.829 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.531 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:32.533 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:33.534 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:34.535 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:34.829 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:34.830 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:34.830 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:34.830 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:34.831 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:34.831 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:34.831 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:34.831 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:34.831 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.831 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:34.831 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.831 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.831 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:34.831 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.537 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:36.538 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:37.540 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:38.542 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:38.831 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:38.832 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:38.832 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:38.833 7f1b5ae7bea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.543 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:40.546 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:41.547 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:42.548 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:42.833 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:42.834 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:42.834 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:42.834 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:42.834 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:42.835 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:42.835 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:42.835 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:42.835 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.835 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:42.835 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.835 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.835 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:42.835 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.549 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:44.551 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:45.552 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:46.554 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:46.835 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:46.836 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:46.836 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:46.836 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:46.836 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:46.836 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:46.837 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:46.837 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:46.837 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.837 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:46.837 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.837 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.837 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:46.837 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.556 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:48.558 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:49.559 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:50.561 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:50.836 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:50.838 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:50.838 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:50.838 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:50.838 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:50.838 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:50.838 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:50.838 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:50.838 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.839 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:50.839 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.839 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.839 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:50.839 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.563 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:52.565 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:53.566 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:54.567 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:54.838 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:54.840 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:54.840 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:54.840 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:54.840 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:54.840 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:54.841 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:54.841 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:54.841 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.841 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:54.841 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.841 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.841 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:54.841 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.569 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:56.570 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:57.572 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:58.574 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:58.841 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:58.842 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:58.842 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:58.842 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:58.842 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:58.842 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:58.842 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:58.843 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:58.843 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.843 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:58.843 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.843 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.843 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:58.843 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.575 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:00.577 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:01.579 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:02.580 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:02.842 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:02.844 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:02.844 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:02.844 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:02.844 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:02.844 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:02.845 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:02.845 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:02.845 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.845 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:02.845 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.845 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.846 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:02.846 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.582 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:04.583 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:05.585 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:06.587 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:06.845 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:06.846 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:06.847 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:06.847 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:06.847 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:06.847 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:06.847 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:06.847 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:06.847 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.847 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:06.847 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.847 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.848 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:06.848 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.588 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:08.589 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:09.590 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:10.592 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:10.847 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:10.848 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:10.848 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:10.848 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:10.849 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:10.849 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:10.849 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:10.849 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:10.849 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.849 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:10.849 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.849 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.849 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:10.849 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.593 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:12.595 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:13.597 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:14.340 7f1b5af812c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:14:14.341 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.341 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:14:14.341 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.598 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:14.849 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:14.850 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:14.850 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:14.850 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:14.850 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:14.850 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:14.850 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:14.851 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:14.851 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.851 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:14.851 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.851 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.851 7f1b5ae7bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:14.851 7f1b5ae7bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.056 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=26) 09/01/2025 02:14:15.057 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=26) is either full or getting a new batch 09/01/2025 02:14:15.057 7f1b5af81a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:14:15.057 7f1b5af81a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:14:15.057 7f1b5af81a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=26) 09/01/2025 02:14:15.058 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {3} 09/01/2025 02:14:15.058 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {3: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x0f\x11}\x92\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\r\xb9\x11\x00\x00\x01\x99\x03\r\xb9\x11\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:14:15.058 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x0f\x11}\x92\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\r\xb9\x11\x00\x00\x01\x99\x03\r\xb9\x11\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:14:15.058 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x0f\x11}\x92\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\r\xb9\x11\x00\x00\x01\x99\x03\r\xb9\x11\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:14:15.058 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Request 10 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x0f\x11}\x92\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\r\xb9\x11\x00\x00\x01\x99\x03\r\xb9\x11\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:14:15.059 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 119916.972656, idle connection 359914.000000, request 29999.952555 09/01/2025 02:14:15.059 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.992619, metadata 119916.506104, idle connection 359913.000000, request 29999.494553 09/01/2025 02:14:15.062 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Received correlation id: 10 09/01/2025 02:14:15.062 7f1b5ae7b540 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:14:15.062 7f1b5ae7b540 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.131', 9092)]>: Response 10 (3.9052963256835938 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:14:15.063 7f1b5ae7b540 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:14:15.063 7f1b5ae7b540 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=26) with base offset 0 log start offset 0. 09/01/2025 02:14:15.063 7f1b5ae7b540 [kafka.client] [DEBUG]: Timeouts: user 9999999989.993334, metadata 119912.649414, idle connection 359909.000000, request inf 09/01/2025 02:14:15.063 7f1b5af81a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:14:15.600 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:16.601 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:17.603 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:18.604 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:18.851 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:18.852 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:18.852 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:18.852 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:18.852 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:18.852 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:18.852 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:18.853 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:18.853 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.853 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:18.853 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.853 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.853 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:18.853 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.605 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:20.607 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:21.608 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:22.610 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:22.853 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:22.854 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:22.854 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:22.854 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:22.854 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:22.854 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:22.854 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:22.855 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:22.855 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.855 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:22.855 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.855 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.855 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:22.855 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.612 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:24.613 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:25.615 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:26.616 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:26.854 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:26.856 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:26.856 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:26.856 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:26.856 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:26.856 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:26.856 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:26.856 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:26.856 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.857 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:26.857 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.857 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.857 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:26.857 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.617 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:28.619 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:29.620 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:30.622 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:30.856 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:30.857 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:30.857 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:30.857 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:30.857 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:30.857 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:30.857 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:30.858 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:30.858 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.858 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:30.858 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.858 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.858 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:30.858 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.623 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:32.624 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:33.625 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:34.626 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:34.857 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:34.858 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:34.858 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:34.858 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:34.858 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:34.858 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:34.858 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:34.859 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:34.859 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.859 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:34.859 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.859 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.859 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:34.859 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.628 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:36.630 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:37.632 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:38.634 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:38.858 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:38.860 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:38.860 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:38.860 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:38.860 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:38.860 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:38.860 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:38.860 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:38.860 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.861 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:38.861 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.861 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.861 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:38.861 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.635 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:40.637 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:41.638 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:42.640 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:42.860 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:42.862 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:42.862 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:42.862 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:42.862 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:42.862 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:42.862 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:42.863 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:42.863 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.863 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:42.863 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.863 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.863 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:42.863 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.642 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:44.643 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:45.644 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:46.646 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:46.863 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:46.864 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:46.864 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:46.864 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:46.864 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:46.865 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:46.865 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:46.865 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:46.865 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.865 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:46.865 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.865 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.865 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:46.865 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.648 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:48.650 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:49.651 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:50.653 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:50.865 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:50.866 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:50.866 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:50.866 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:50.866 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:50.866 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:50.866 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:50.867 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:50.867 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.867 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:50.867 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.867 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.867 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:50.867 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.654 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:52.656 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:53.658 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:54.658 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:54.867 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:54.868 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:54.868 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:54.868 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:54.868 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:54.868 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:54.868 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:54.869 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:54.869 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.869 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:54.869 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.869 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.869 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:54.869 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.660 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:56.661 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:57.663 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:58.664 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:58.868 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:58.869 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:58.870 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:58.870 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:58.870 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:58.870 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:58.870 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:58.870 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:58.870 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.871 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:58.871 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.871 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.871 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:58.871 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.665 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:00.666 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:01.305 7f1b5ae7ba40 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:01.668 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:02.670 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:02.870 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:02.871 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:02.871 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:02.871 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:02.871 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:02.872 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:02.872 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:02.872 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:02.872 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:02.872 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:02.872 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:02.872 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:02.872 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:02.872 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.671 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:04.672 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:05.674 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:06.675 7f1b5af81540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:06.872 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:06.873 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:06.873 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:06.874 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:06.874 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:06.874 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:06.874 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:06.874 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:06.874 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:06.874 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:06.874 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:06.874 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:06.875 7f1b5ae7ba40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756691954324159 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:15:06.875 7f1b5ae7ba40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle]